Does blocking a page in robots.txt remove it from search results?
Not necessarily. robots.txt stops crawling, not indexing. If other sites link to the page, its URL can still appear in results with no description, because the crawler was never allowed to read it. To take a page out of results reliably, leave it crawlable and add noindex (a meta tag or an X-Robots-Tag header). Blocking it in robots.txt actually keeps it in, because the crawler can never see the noindex.
If Allow and Disallow both match, which wins?
Under RFC 9309, which Google and Bing follow, the rule with the longest matching path wins. Given Disallow: /private/ and Allow: /private/ok, the path /private/ok is allowed because the Allow matches 10 characters against the Disallow's 9, while /private/x only matches the Disallow and is blocked. When two rules match with exactly the same length, the more permissive Allow wins. None of this depends on the order the lines appear in the file.
How do * and $ work?
* matches any run of characters, so Disallow: /*? covers every URL containing a question mark and Disallow: /*.pdf covers every URL containing .pdf. $ is special only at the end of a rule, where it anchors the end of the URL: Disallow: /*.pdf$ blocks URLs ending in .pdf but leaves /a.pdf?x=1 alone. Neither is in the original 1994 specification, but every major search engine supports them.
What happens if robots.txt returns 404?
A 404 means "no restrictions", and crawlers proceed to crawl the whole site — not having a robots.txt is perfectly fine. Persistent 5xx errors are different: Google may err on the safe side and temporarily treat the entire site as disallowed. So if you do not want a robots.txt, return a clean 404 rather than a server error or a redirect loop.