Server authors do not write robots.txt files with social link previews in mind, because no other social media software has used robots.txt to ...
Fetch robots.txt (and cache it) for that remote URL. Link previews should respect robots.txt #21738; Lazily fetch the preview only when it's ...
Old Hard to Find TV Series on DVD
Description When we send a link in Matrix, the client can use the Synapse's integrated prefetcher to fetch the link preview.
Store images and toots on IPFS to reduce inter-instance traffic and local caching requirements #21461.
GitHubLink previews should respect robots.txt · Issue #21738 · mastodon/mastodonSteps to reproduce the problem Share https://www.jefftk.com/test/no-robots or ...
@tw @crschmidt @cshabsin link preview bots all ignore robots.txt, so ... Link previews should respect robots.txt · Issue #21738 · mastodon/mastodonGitHub.
Saved shot preview image and details are no longer displayed, only the saved shot link. [Notes]:. Here is a screenshot with the issue:
I propose to use Robots Parser library with this common functions in utils.js : getRobotsTxt(url); isAllowedRobotsTxt(url, ua) ...
Connect will serve a default robots.txt that disallows all crawling. Admins can specify a custom robots.txt using the Branding.Robots configuration option.
Fixed robots.txt 404 error (#2226); Fix counts on issues dashboard (#2215); Fix unclosed session bug (#2214); Add collaborative repositories to the dashboard ...