htdig Not Indexing Site
I installed ht://Dig 3.2.0b5 on one of my Solaris 10 servers. When I ran htdig
on the server, it did not appear to be indexing my website. I used
/usr/localbin/rundig -s -c /usr/local/conf/htdig_support.conf to
see statistics on what it was doing. It was only opening one connect and making
just two HTTP requests rather than indexing the whole site. When I ran
htdig -vvv, I could see that it was stopping after reading
robots.txt. When I looked at robots.txt, it appeared to be configured to allow
any robot to index all files on the website. It had only the two lines below:
User-agent: *
Disallow:
After experimentation, I found that if I specifed some value for "Disallow",
I could get htdig to index the site. I put in a dummy value, i.e.
Disallow: /abcde12345, a directory
I would never actually use on the site to resolve the problem.
[ More Info ]
[/os/unix/solaris]
permanent link
Finding Hard Links and Symbolic Links
On a Unix or Linux system, you can find symbolic links by utilizing
options with the
find command. To find symbolic links,
aka symlinks, use
find <path> -type l.
E.g.
find / -type l will find every symbolic link on
the system. To find hard links, you can use
find <path> -type
f -links +1. The
-links +1 option tells find to look
for files with more than one link to them. E.g.
find / -type f -links
+1 would search for every hard link on the system.
If you just want to find all symbolic links pointing to a particular file,
e.g. search.html, you can use the find command with the -lname
option.
# find / -lname 'search.html' 2>/dev/null
/usr/share/htdig/index.html
In the above example, the -lname option tells find to look
only for symbolic links to a file named search.html.
Using 2>/dev/null discards
error messages by sending them to /dev/null. Otherwise, you could
a lot of " No such file or directory" messages as well as the symbolic link
information for which you are looking.
If you wish to see full details returned regarding the file, you can use
the -ls option.
# find / -lname 'search.html' -ls 2>/dev/null
146567 0 lrwxrwxrwx 1 root root 11 Nov 8 2003 /usr/share/htdig/index.html -> search.html
References:
-
Using find to locate files
Mo Budlong's UNIX 101 Sunworld column
-
Ln - LQWiki
May 26, 2006
[/os/unix/commands]
permanent link
