Extract Data From Youtube with `yt-dlp` And `jq`

In this case, we have to expand the previous command by generating a JSON array this time by traversing the playlist. This can be done with the following command:

cat barbero.json | jq 'del(..|nulls) | [.entries[] | {show_url: .webpage_url, show_title: .title, show_upload_date: .upload_date, show_duration: .duration, show_description: .description, show_is_live: .is_live, show_origin_automatic_caption: .automatic_captions | with_entries(if (.key|test(".*orig")) then ( {key: .key, value: .value } ) else empty end)[][] | select(.ext|contains("json")) | .url }]'

The highlights here are:

• del(..|nulls): that eliminates all the null values in the JSON. Necessary as we will get an error if we are gonna try to extract data from a null value.
• The outer square parenthesis. If omitted we will get a series of JSON objects one after the other, but not a valid JSON.
• The traverse of an array value (eg. .entries[]). With the [] at the end it means: compute all the entries from now on.
• with_entries(if (.key|test(".*orig")) then ( {key: .key, value: .value } ) else empty end) it's quite complicated, but basically, it's a way to filter the fields by matching a regexp. Only the field ending with orig will be considered. We are interested in the automatic caption of the original language only.
• select(.ext|contains("json")) filter the entries whose .ext field value contains the word json. We are not interested in other formats, only on JSON.

A YouTube Channel's JSON adds one more layer to the previous case because we might have multiple "playlists", one for:

• Videos
• Shorts
• Live

Therefore, we need to inspect the file and get to the right playlist before applying the previous transformation. The resulting command is:

cat youtubo.json | jq 'del(..|nulls) | .entries[] | select(.title|contains("Videos")) | [.entries[] | {show_url: .webpage_url, show_title: .title, show_upload_date: .upload_date, show_duration: .duration, show_description: .description, show_is_live: .is_live, show_origin_automatic_caption: .automatic_captions | with_entries(if (.key|test(".*orig")) then ( {key: .key, value: .value } ) else empty end)[][] | select(.ext|contains("json")) | .url }]'

You can see at the beginning of the jq command an additional .entries[] step followed by a filter by the values of the .title. That brings up the Videos playlist. The rest of the command is the same as above. If we want to focus on other playlists we just have to filter by a different category.

Once we can get the URL for the automatic captions and retrieve the JSON, we need to extract the text from it. Here is an example of the automatic caption's JSON

The command to do that is the following:

cat f.txt | jq '[.events[] | select(.segs != null) | .segs[] | .utf8]'

We still want a resulting JSON array, but the events should contain some text (the select clause filter). Then we are interested only in that field ignoring the rest.