EON: zpool.cache not getting update

[Manojav Sridhar]

my zpool.cache doesn't seem to be getting updated. How force to persist this information. As a result some zfs set parameters are getting lost on a reboot. Any ideas? there is plenty of space on my usb disk etc. so it cant be that.

[Andre Lue]

That doesn't sound like the zpool.cache

Zfs set parameters are stored on zpool members

What zfs set parameters or command are you running that's not being saved?

[Manojav Sridhar]

i have

zpool set sharenfs=off pool

zfs set sharenfs=on pool/dataset1
zfs set sharenfs=on pool/dataset2

the nfs share's (exports) dont come up on a reboot and i have to explicitly set them on again.

[Andre Lue]

Vajonam,

Can you kindly post the output of below before and after a reboot?
1)ptree
2)zfs get sharenfs pool/dataset1
3)sharemgr show -vp

[Andre Lue]

Vajonam,

I think I know what is going on. The properties will still be set correctly but you'll be missing the /etc/dfs/sharetab entries made when you run zfs set sharenfs=on pool/dataset1

I think adding /etc/dfs/sharetab to /mnt/eon0/.backup and running updimg.sh should solve the issue.

[Manojav Sridhar]

dre2kse,

i've done what you said. I did one reboot and sharetab was not restored. I just rechecked my .backup and tried again this time uncommenting the file printing.

Updating files in /mnt/eon0/.backup to x86.eon
backup in /mnt/eon0/boot/x86.eon.1
/mnt/eon0/.backup: OK
gzcat /mnt/eon0/boot/x86.eon > /tmp/x86.1606
lofiadm -a /tmp/x86.1606 /dev/lofi/1
mounting ... /dev/lofi/1 /mnt/upd
FILE/DIR: /etc/hostid
FILE/DIR: /etc/ipf/ipf.conf
FILE/DIR: /etc/ipf/ipnat.conf
FILE/DIR: /etc/hosts.allow
FILE/DIR: /etc/hosts.deny
FILE/DIR: /etc/group
FILE/DIR: /etc/passwd
FILE/DIR: /etc/shadow
FILE/DIR: /etc/system
FILE/DIR: /etc/syslog.conf
FILE/DIR: /etc/nodename
FILE/DIR: /etc/dhcp.*
FILE/DIR: /etc/dhcpd.*
FILE/DIR: /etc/hostname.e1000g0
FILE/DIR: /etc/ima.conf
FILE/DIR: /etc/pam.conf
FILE/DIR: /etc/resolv.conf
FILE/DIR: /etc/nsswitch.conf
FILE/DIR: /etc/rndc.key
FILE/DIR: /etc/rtc_config
FILE/DIR: /etc/nfssec.conf
FILE/DIR: /etc/dumpadm.conf
FILE/DIR: /etc/power.conf
FILE/DIR: /etc/rsyncd.conf
FILE/DIR: /etc/rsyncd.secrets
FILE/DIR: /etc/defaultrouter
FILE/DIR: /etc/defaultdomain
FILE/DIR: /etc/default/init
FILE/DIR: /etc/default/login
FILE/DIR: /etc/default/nfs
FILE/DIR: /etc/dfs/dfstab
FILE/DIR: /etc/dladm/datalink.conf
FILE/DIR: /etc/dladm/flowadm.conf
FILE/DIR: /etc/dladm/flowprop.conf
FILE/DIR: /etc/dladm/secobj.conf
FILE/DIR: /etc/zfs/zpool.cache
FILE/DIR: /etc/ssh/ssh_config
FILE/DIR: /etc/ssh/ssh_host_dsa_key
FILE/DIR: /etc/ssh/ssh_host_dsa_key.pub
FILE/DIR: /etc/ssh/ssh_host_rsa_key
FILE/DIR: /etc/ssh/ssh_host_rsa_key.pub
FILE/DIR: /etc/ssh/sshd_config
FILE/DIR: /etc/profile
FILE/DIR: /etc/.bash_profile
FILE/DIR: /etc/inet/hosts
FILE/DIR: /etc/inet/ntp.conf
FILE/DIR: /etc/inet/ntp.drift
FILE/DIR: /etc/krb5/krb5.keytab
FILE/DIR: /etc/krb5/krb5.conf
FILE/DIR: /etc/user_attr
FILE/DIR: /etc/mail/sendmail.cf
FILE/DIR: /.ssh/*
FILE/DIR: /.sunw/*
FILE/DIR: /.profile
FILE/DIR: /root/.ssh
FILE/DIR: /root/.sunw
FILE/DIR: /root/.kshrc
FILE/DIR: /root/.profile
FILE/DIR: /admin/.ssh
FILE/DIR: /admin/.kshrc
FILE/DIR: /admin/.profile
FILE/DIR: /admin/.bash_profile
FILE/DIR: /export/home/admin/.ssh
FILE/DIR: /export/home/admin/.kshrc
FILE/DIR: /export/home/admin/.profile
FILE/DIR: /export/home/admin/.bash_profile
FILE/DIR: /etc/sfw/smb.conf
FILE/DIR: /etc/sfw/private/*
FILE/DIR: /etc/php/5.2/php.ini
FILE/DIR: /etc/apache2/httpd.conf
FILE/DIR: /etc/apache2/server.crt
FILE/DIR: /etc/apache2/server.key
FILE/DIR: /etc/apache2/2.2/httpd.conf
FILE/DIR: /etc/apache2/2.2/server.crt
FILE/DIR: /etc/apache2/2.2/server.key
FILE/DIR: /etc/apache2/2.2/conf.d/modules-32.load
FILE/DIR: /etc/apache2/2.2/conf.d/modules-64.load
FILE/DIR: /etc/apache2/2.2/conf.d/php5.2.conf
FILE/DIR: /etc/apache2/2.2/conf.d/ssl.conf
FILE/DIR: /etc/apache2/2.2/conf.d/userdir.conf
FILE/DIR: /var/smb/osmbpasswd
FILE/DIR: /var/smb/smbgroup.db
FILE/DIR: /var/smb/smbpasswd
FILE/DIR: /var/samba/*
FILE/DIR: /var/samba/locks/*
FILE/DIR: /var/idmap/idmap.db
FILE/DIR: /usr/bin/setup
FILE/DIR: /usr/bin/install.sh
FILE/DIR: /usr/bin/updimg.sh
FILE/DIR: /lib/svc/method/devices-local
FILE/DIR: /var/spool/cron/crontabs/adm
FILE/DIR: /var/spool/cron/crontabs/root
FILE/DIR: /boot/solaris/bootenv.rc
FILE/DIR: /var/apache2/2.2/cgi-bin/napp-it/napp-it.cfg
FILE/DIR: /root/.ssh/authorized_keys
FILE/DIR: /root/.ssh/config
FILE/DIR: /root/.ssh/id_dsa
FILE/DIR: /root/.ssh/id_dsa.keystore
FILE/DIR: /root/.ssh/known_hosts
FILE/DIR: /etc/dfs/sharetab
copying /etc/svc/repository.db
umounting ... /mnt/upd
lofiadm -d /dev/lofi/1
mv -f /mnt/eon0/boot/x86.eon.0 /mnt/eon0/boot/x86.eon.1
mv -f /mnt/eon0/boot/x86.eon /mnt/eon0/boot/x86.eon.0
gzip -f -9 -c /tmp/x86.1606 > /mnt/eon0/boot/x86.eon
/mnt/eon0/boot/x86.eon: OK

its there at the end, but after the reboot, the zfs set comman-ed shares are still missing its like its been overwritten after the bootup. any other ideas?

[Manojav Sridhar]

I have attached the output of the commands you requested.

getafix:4:~#ptree
7 /lib/svc/bin/svc.startd
866 /usr/lib/saf/ttymon -g -d /dev/console -l console -m ldterm,ttcompat -h
9 /lib/svc/bin/svc.configd
18 /sbin/dlmgmtd
245 /usr/lib/idmapd
249 /lib/crypto/kcfd
252 /usr/lib/sysevent/syseventd
255 devfsadmd
262 /usr/lib/sysevent/syseventconfd
709 /lib/svc/bin/svc.ipfd
711 /usr/sbin/ipmon -Ds
759 /usr/sbin/cron
772 /usr/lib/smbsrv/smbd start
832 /usr/sbin/rpcbind
860 /usr/sbin/syslogd
864 /usr/lib/utmpd
869 /usr/lib/ssh/sshd
1193 /usr/lib/ssh/sshd
1194 /usr/lib/ssh/sshd
1197 -bash
1213 ptree
940 /usr/apache2/2.2/bin/httpd -k start
951 /usr/apache2/2.2/bin/httpd -k start
952 /usr/apache2/2.2/bin/httpd -k start
953 /usr/apache2/2.2/bin/httpd -k start
941 /usr/sbin/nscd
944 /usr/lib/inet/ntpd -p /var/run/ntp.pid -g
960 /usr/lib/nfs/statd
969 /usr/lib/nfs/lockd
1150 /usr/lib/nfs/mountd
1152 /usr/lib/nfs/nfsd
1188 /usr/bin/rsync --daemon
getafix:5:~# zfs get sharenfs kalahari/downloads
NAME PROPERTY VALUE SOURCE
kalahari/downloads sharenfs on local
getafix:6:~#sharemgr show -vp
default nfs=()
zfs
zfs/atacama nfs=()
/atacama
zfs/atacama/media nfs=() smb=()
atacama=/atacama/media
zfs/kalahari nfs=()
/kalahari
zfs/kalahari/downloads nfs=() smb=()
downloads=/kalahari/downloads
zfs/kalahari/media nfs=() smb=()
kalahari=/kalahari/media
zfs/kalahari/shared nfs=() smb=()
shared=/kalahari/shared
zfs/sahara nfs=()
/sahara
zfs/sahara/media nfs=() smb=()
sahara=/sahara/media

getafix:7:~#cat /etc/dfs/sharetab
/sahara/media - smb ""
/atacama/media - smb ""
/sahara/media - nfs rw
/sahara - nfs rw
/atacama/media - nfs rw

[Manojav Sridhar]

any ideas dre2kse?

Message was edited by: vajonam

[Andre Lue]

Hi Vajonam,

I'm not sure what's going on so I would have to try to re-create your setup for closer inspection. This will take some time. From what I see I can tell everything is being preserved as designed.
-zpool information shows sharenfs still on
-/etc/dfs/sharetab is being preserved (with kalahari shares)

There is some inconsistency in the output, Can you clarify? For example a kalahari nfs share in sharemgr but not in sharetab.

Can you paste the output of /etc/vfstab?

[Manojav Sridhar]

dre2kse,

here is my /etc/vfstab

#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
/devices - /devices devfs - no -
/proc - /proc proc - no -
ctfs - /system/contract ctfs - no -
objfs - /system/object objfs - no -
sharefs - /etc/dfs/sharetab sharefs - no -
fd - /dev/fd fd - no -
swap - /tmp tmpfs - yes -

/devices/ramdisk:a - / ufs - no nologging


My point is that /etc/dfs/sharetab is being preserved but then is overwritten by some process, I haven't been able to find out which process overwrites it.

[Andre Lue]

Hi Vajonam,

I ran thru the steps to enable and preserve nfs and it seems to be working after a reboot.

Can you detail/clarify what is meant by /etc/dfs/sharetab is getting overwritten? Is that happening repeatedly on a reboot? if so can you make a before and after copy?

Please feel free to share any other details you think may help. If possible share the steps you ran to get to this point.

[Manojav Sridhar]

Well,

I have the following

sharetab before I run updimg.sh

getafix:3:~#cat /etc/dfs/sharetab
/sahara/media - smb ""
/kalahari/downloads - smb ""
/kalahari/media - smb ""
/atacama/media - smb ""
/sahara/media - nfs rw
/sahara - nfs rw
/kalahari/downloads - nfs rw
/kalahari/media - nfs rw
/atacama/media - nfs rw

then i run updimg.sh, to back it up. then reboot

/sahara/media - smb ""
/atacama/media - smb ""
/sahara/media - nfs rw
/sahara - nfs rw
/atacama/media - nfs rw


but only the last two shares i have added kalahari/* aren't on the sharetab, others show up fine. what is the process that does this mounting / populating the sharetab?

to get it work after a reboot I have to run

getafix:2:~#zfs set sharenfs=on kalahari/downloads
getafix:3:~#zfs set sharenfs=on kalahari/media
getafix:4:~#cat /etc/dfs/sharetab
/sahara/media - smb ""
/kalahari/downloads - smb ""
/kalahari/media - smb ""
/atacama/media - smb ""
/sahara/media - nfs rw
/sahara - nfs rw
/kalahari/downloads - nfs rw
/kalahari/media - nfs rw
/atacama/media - nfs rw

after which all is good again

How can i check what is actually in the saved image (as a result of updimg.sh)?

[Andre Lue]

I have not seen this before. To check what is saved in the image you can simply boot the new image after updimg.sh and look what is in /etc/dfs/sharetab or

gzcat /path/image > /some/path/ungzipped_image
lofiadm -a /some/path/ungzipped_image /dev/lofi/1
mount /dev/lofi/1 /some/mnt
cd /some/mnt/etc/dfs
cat sharetab

[Manojav Sridhar]

dre2kse,

thanks for that info, i've had a look at the image and it does contain all the right information in the sharetab.

so somewhere after the unzip of the image to the starting up of the zfs stuff this gets overwritten.

is is possible to watch that file for changes? any ideas on how I can accomplish this? the changes i made reccently was enable swap in the .exec, which i have tried to back out with no difference. I also enabled the nfs/mapid service to enable NFS4 from linux clients.

the last resort is to add zfs export/import commands to the end of the .exec, i would to find a solution before i try that, any other ideas?

[Andre Lue]

Hi Vajonam,

Based on what you describe, it sounds more like the zpool for some reason not keeping the state of the last 2 sharenfs cmds. Can you paste the output of zpool status -v

To disable the swap from mounting, simply commenting it in .exec should work. The swap filesystem will still exist but will not be mounted. A Zfs swap is highly recommended.

Try commenting zpool import line in .exec and reboot and tell me what sharetab shows.

[Manojav Sridhar]

right, which is how this whole thread got started, where i suspected the zpool.cache was not reflecting what was actually in the pool. not sure if that is the case.

here is the command after a reboot.

getafix:2:~#zpool status -v
pool: atacama
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
atacama ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0

errors: No known data errors

pool: kalahari
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
kalahari ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0

errors: No known data errors

pool: sahara
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
sahara ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c2t2d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0

errors: No known data errors

Other questions.

the /etc/dfs/sharetab is empty when i comment out the zpool import -a from the exec

then when i run it from the command line i get.

getafix:3:~#zpool status
no pools available
getafix:4:~#zpool import -a
cannot share 'atacama': share(1M) failed
cannot share 'kalahari': share(1M) failed
cannot share 'kalahari/downloads': share(1M) failed
cannot share 'kalahari/media': share(1M) failed
cannot share 'kalahari/shared': share(1M) failed

what does that mean?

[Andre Lue]

Vajonam,
[quote]
right, which is how this whole thread got started, where i suspected the zpool.cache was not reflecting what was actually in the pool. [/quote]

zpool.cache does not hold that type of info. Those are part of the ON DISK specification and kept on actual pool members. See pages 35,36

http://hub.opensolaris.org/bin/download/Community+Group+zfs/docs/ondiskformat0822.pdf

What type of disk is each member in atacama and kalahari?
Are they USB attached disks?

Can you run the following and post dump.cache and /mnt/eon0/zpool.cache
zdb -C > /tmp/dump.cache

[Manojav Sridhar]

dre2kse,

the disks are all SATA, atacama's RAID1 attached ICH8 controller. kalahari are to SATA connected to a LSI RAID all in JBOD config, and using ZFS for mirror.

zdb dump is at is at

http://pastebin.ca/1912358

what does the "cannot share" point to? some nfs server not running at the time? maybe I will try to disable my mapid service to see if this affected anything.

thanks

[Andre Lue]

I'm not sure what to make "cannot share" "share failed" yet.

Can you post back the output for

share -F nfs atacama

share -F nfs kalahari/downloads

also after running those /var/svc/log/network-nfs-server:default.log

[Manojav Sridhar]

will post that info shortly

[Manojav Sridhar]

dre2kse,

here are output that you requeted.

getafix:1:~#zpool import -a

getafix:2:~#cat /etc/dfs/sharetab
/sahara/media - smb ""
/atacama/media - smb ""
/sahara/media - nfs rw
/sahara - nfs rw
/atacama/media - nfs rw


getafix:5:~#cd /
getafix:6:/#share -F nfs atacama
getafix:7:/#share -F nfs kalahari/downloads

getafix:8:/#cat /var/svc/log/network-nfs-server:default.log
[ Aug 23 08:11:21 Executing start method ("/lib/svc/method/nfs-server start"). ]
[ Aug 23 08:11:24 Method "start" exited with status 0. ]

[Manojav Sridhar]

dre2kse,

any chance you have run into this? i still have this issue, but have since not done too many reboots so its not a day to day issue, but will be nice to get sorted..

thanks

[Andre Lue]

No, I have not been able to replicate nor have I run into this issue.

[Manojav Sridhar]

so, i had some time to go and re-investigate this.

instead of using zpool import -a, if i specify the pools i wanted to import in a particular order, all the shares and everything else came back correctly.

zpool import sahara "2nd pool"
zpool import atacama "first pool"
zppol import kalahari "3rd pool"

things work.. maybe this helps ? i saw some bugs with zpool import order, but i though they were unrelated.