[FFmpeg-user] Output to S3

[Bohdan Fihar]

Hi Edward,

No, my C# code doesn't contain anything related ASW S3, it just sends a
file to Url:
        private const string filePath =
@"d:\Programs\ffmpeg-20200831-4a11a6f-win64-static\bin\01.05.flv";

        private static void UploadObject(string url)
        {
            var httpRequest = System.Net.WebRequest.Create(url) as
System.Net.HttpWebRequest;
            httpRequest.Method = "PUT";
            using (var dataStream = httpRequest.GetRequestStream())
            {
                var buffer = new byte[8000];
                using (var fileStream = new System.IO.FileStream(filePath,
System.IO.FileMode.Open, System.IO.FileAccess.Read))
                {
                    int bytesRead = 0;
                    while ((bytesRead = fileStream.Read(buffer, 0,
buffer.Length)) > 0)
                    {
                        dataStream.Write(buffer, 0, bytesRead);
                    }
                }
            }
            var response = httpRequest.GetResponse() as
System.Net.HttpWebResponse;
        }

No credentials required, since it's a public bucket.
By the way, does FFMPEG support sending to HttpS ?

I tried adding these parameters*:-send_expect_100 true* and *-multiple_requests
false,* Also I added Content-Length header:
ffmpeg -i sample_640x360.ts -send_expect_100 true  -multiple_requests false
 -loglevel trace  -method PUT -headers "Content-Length: 646720"
https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts

The result is:
[tcp @ 00000205536a5340] Starting connection attempt to 52.219.84.232 port
443
[tcp @ 00000205536a5340] Successfully connected to 52.219.84.232 port 443
[https @ 00000205536a5000] request: PUT /video/output.ts HTTP/1.1
Transfer-Encoding: chunked
User-Agent: Lavf/58.51.101
Accept: */*
Expect: 100-continue
Connection: close
Host: bogdan-public.s3.us-east-2.amazonaws.com
Icy-MetaData: 1
Content-Length: 646720

[https @ 00000205536a5000] header='HTTP/1.1 501 Not Implemented'
[https @ 00000205536a5000] http_code=501
[https @ 00000205536a5000] HTTP error 501 Not Implemented
https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts: Server
returned 5XX Server Error reply
[AVIOContext @ 00000205536a7100] Statistics: 577680 bytes read, 2 seeks

Looks like something is missing.


On Wed, 9 Sep 2020 at 12:41, Edward Park <kumowoon1025 at gmail.com> wrote:

> Hi,
>
> >> Is setting up a sort of proxy server that will intermediate and upload
> >> to s3 an option?
> > It might be the option, but it will be the last option I'd like to
> accept.
> > I'm seeking for more simple solution.
> > I'm afraid that I'll stuck on sending a video to that server too.
> Is the aversion to using the official s3 utilities due to resources and
> lack of access, or the additional integration and associated learning curve
> that it brings? If it's the latter, and you can configure the server to
> your needs, I think you might find any of the solutions that mount
> s3-compatible buckets as network shares, or even local fuse filesystems to
> your liking. Performance is not the best, and you will see request graph
> spike like mad, but it is in my opinion the most simple solution. (Since
> the s3 storage will be just like any other local path on the machine) Worth
> some consideration imo.
>
> >> That is a very small window of view into the error you got, I'd try
> >> using the send_expect_100 option and confirming which ssl library is
> used,
> >> no real basis but what I might start with when I do the throw all of it
> on
> >> the wall and see what sticks routine.
> > The error says:
> > Error in the push function.
> > av_interleaved_write_frame(): I/O error
> > Error writing trailer of
> > https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts: I/O
> error
> > frame=  218 fps=184 q=10.9 Lsize=     404kB time=00:00:07.17 bitrate=
> > 461.4kbits/s speed=6.06x
> > video:352kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB
> > muxing overhead: 14.843175%
> > [tls @ 000001b41de52940] The specified session has been invalidated for
> > some reason.
> > [tls @ 000001b41de52940] Error in the pull function.
> > [https @ 000001b41de52840] URL read error: I/O error
> > Conversion failed!
> I am thinking that the "Error in the push function" is not from FFmpeg. I
> missed it earlier, the quoting was messed up in my email client, but the
> only problem FFmpeg seems to be reporting is "I/O error," and consistently
> before or after sending the main data body. (Let me know if I am wrong
> about this)
>
> So in addition to the -send_expect_100 true, I would add
> -multiple_requests false to disable pipelining
>
> > This is the whole output:
> >
> > d:\Programs\ffmpeg-20200831-4a11a6f-win64-static\bin>
> > ffmpeg -i sample_640x360.ts -method PUT
> > https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts
> But the problem is the invalidated session, it suggests that you
> successfully got a valid session authorized beforehand, if it was
> invalidated there's no way to know why unless we start from there. Is this
> command unabridged as well? I think you mentioned you altered the url, but
> there's no api key or anything I would expect to be in the request header
> that uploads a file to an endpoint.
>
> > I tried several video formats with this command:
> > ffmpeg -i sample_640x360.ts -method PUT
> > https://bogdan-public.s3.us-east-2.amazonaws.com/video/output.ts
> >
> > The bucket (its name is changed here) is public and I was able to send
> data
> > to it by Http Put request from the C# code sample.
>
> I have a feeling the C# code sample was part of a whole AWS S3 C# SDK,
> handling all AAA with credentials hardcoded in a configuration file or
> something. Doing the same with a REST api will require more hands-on
> operation.
>
> Regards,
> Ted Park
>
> _______________________________________________
> ffmpeg-user mailing list
> ffmpeg-user at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".



-- 
All the best,

Bohdan