Skip to content

feat(twitter): resolve reposts, expand t.co links and resume tweets pagination - #2518

Open
UncertaintyDeterminesYou4ndMe wants to merge 1 commit into
jackwener:mainfrom
UncertaintyDeterminesYou4ndMe:feat/twitter-tweets-retweet-urls-cursor
Open

UncertaintyDeterminesYou4ndMe wants to merge 1 commit into
jackwener:mainfrom
UncertaintyDeterminesYou4ndMe:feat/twitter-tweets-retweet-urls-cursor

Conversation

@UncertaintyDeterminesYou4ndMe

Copy link
Copy Markdown

Description

Three additive capabilities for opencli twitter tweets, motivated by backfilling a user's timeline (thousands of reposts) into a local Markdown archive:

  1. retweeted_tweet column — a repost row only carries the truncated "RT @user: …" string, the reposter as author, and no media. The original post is now surfaced from legacy.retweeted_status_result in the same shape as the existing quoted_tweet (id, author, name, full text, counts, created_at, url, media, and its own quoted_tweet). Tombstoned / unavailable originals yield null; is_retweet is unchanged.
  2. --expand-urls (default false) — replaces https://t.co/… in text, quoted_tweet.text and retweeted_tweet.text with expanded_url, collected from legacy.entities.urls, the long-form note_tweet … entity_set.urls (links in long posts are only listed there) and the media entities (the trailing t.co of attached photos/videos).
  3. --cursor / --cursor-file — resume pagination from a saved bottom cursor and persist the next one (empty file = timeline exhausted). Together with the existing --page-delay this makes gentle, resumable backfills possible without re-fetching pages or tripping the 429 limiter. Details:
    • with --cursor-file, --limit becomes a soft target: whole pages are returned so no tweet falls between two runs (cursors point at page boundaries);
    • if the first page of a run fails (e.g. HTTP 429) the resume point is written before the error is raised, so the caller can retry the same page after cooldown;
    • a resumed run that finds no tail returns [] instead of EmptyResultError.

Output is unchanged when the new flags are off, apart from the appended retweeted_tweet column. collectTweetUrlMap, expandTweetUrls and extractRetweetedTweet live in shared.js so timeline / search / list-tweets can adopt them later.

opencli twitter tweets @jack --limit 500 --expand-urls true --page-delay 4 --cursor-file ./jack.cursor -f json
# later: continue from where the previous run stopped
opencli twitter tweets @jack --limit 500 --cursor "$(cat ./jack.cursor)" --cursor-file ./jack.cursor -f json

Related issue: —

Type of Change

  • ✨ New feature

Checklist

  • I ran the checks relevant to this PR — npx vitest run --project adapter clis/twitter/ (560 passed) and the default gate npm test; the only failures are 6 pre-existing ones in src/cli.test.ts (browser tab targeting) that fail identically on an untouched main checkout in my environment.
  • I updated tests or docs if needed — new cases in tweets.test.js (expand-urls, retweeted_tweet, tombstone, cursor resume + cursor-file, rate-limited resume, empty resumed tail) and shared.test.js (helpers); usage example added to docs/adapters/browser/twitter.md; cli-manifest.json rebuilt.
  • I included output or screenshots when useful

Documentation

  • Used positional args for the command's primary subject unless a named flag is clearly better
  • Normalized expected adapter failures to CliError subclasses instead of raw Error

Screenshots / Output

Live run against my own logged-in account (--limit 5 --expand-urls true --cursor-file ./cur.txt -f json), one row abridged:

{
  "id": "2100136163877208198",
  "author": "snwkng50462474",
  "is_retweet": true,
  "text": "RT @AdrianPunk115: 本文分为上下两册,直接作为做网页动效的指南。 https://x.com/i/article/2098669987930210304",
  "retweeted_tweet": {
    "id": "2100136...",
    "author": "AdrianPunk115",
    "text": "本文分为上下两册,直接作为做网页动效的指南。 https://x.com/i/article/2098669987930210304",
    "url": "https://x.com/AdrianPunk115/status/...",
    "has_media": false
  }
}

./cur.txt afterwards holds the bottom cursor of the fetched page; a second run with --cursor "$(cat cur.txt)" continued from it, and reaching the end of the timeline left the file empty.

… pagination

`twitter tweets` gains three additive capabilities for backfilling a user
timeline into a local archive:

- `retweeted_tweet` column: a repost row only carries a truncated
  `"RT @user: …"` string and the reposter as author. The original post
  (full text, real author, media, counts, and its own `quoted_tweet`) is now
  surfaced from `legacy.retweeted_status_result` in the same shape as
  `quoted_tweet`. Tombstoned / unavailable originals yield `null`.
- `--expand-urls`: replaces `https://t.co/…` links in `text` (and in the
  quoted / retweeted texts) with `expanded_url`, using `legacy.entities.urls`,
  the long-form `note_tweet` entity set, and the media entities.
- `--cursor` / `--cursor-file`: resume pagination from a saved bottom cursor
  and persist the next one (empty file = timeline exhausted). With a cursor
  file, `--limit` becomes a soft target so whole pages are returned and no
  tweet falls between two runs. On a rate-limited first page the resume point
  is kept before the error is raised; a resumed run that finds no tail returns
  `[]` instead of `EmptyResultError`.

Existing output stays byte-identical when the new flags are off (only the new
`retweeted_tweet` column is appended). Helpers live in `shared.js` so other
timeline adapters can reuse them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant