Linkrot resolution and dead end flagging

[TomLucidor]

• When people change their URL address, this can cause issues in the records, there should be ways of capturing these hickups
• In the case where the whole blog is wiped off the internet, InternetArchive and others would be a proper substitute
• RIP in the case of Twitter/X, but there needs to be a way to IA every blog the agent touches (queued to not spam the servers)

But another problem pops up in my head: assuming spam SEO links are a problem, bad URLs need to be flagged, starting with pre-established blocklists, and building up from there

[rmusser01]

Scheduled scraping is planned, along with content detection changes to identify new/changed information.
Had planned to implement this with the simple RSS feed functionality, since it would require some sort of regularity and timing, and then also requires some type of sorting which would tie in with user preference identification/customization. Which would then tie into automatic/scheduled web scraping.

[TomLucidor]

Detect content changes? "Content-Defined Chunking" has some ideas on how to solve it... But dunno how to mash them with tokenization (maybe not at all if multiple models will be used and the same string can yield to different embeddings unless standardized like ModernBRET or Pile-T5/ByT5).
Text preference training can be based on paragraph level or article level data, and using either naively classifying embeddings, or using recommendation engines

[rmusser01]

Simply hash the content and compare hashes

[TomLucidor]

So contextual awareness + hashes that are faster than MD5 but longer (so as to avoid collision by accident)? What if the "ad" or "recommendation" sections change but not the core article? Can it differentiate?