Tuesday, February 7, 2012

Subversion Tricks



Below is my list of commands that I commonly run when working with subversion on my projects. I never can remember them so I come here to copy and paste when I need to...


Delete versioned files recursively using a wildcard match

find ./ -name *.pidb | xargs svn rm --force


Set svn properties recursively from a text file on the file system

svn propset svn:ignore -R -F /svn/ignores.txt .


Revert all changes in folders and files recursively on the file system

svn revert -R .


Remove versioning of files recursively with a wildcard and without deleting locally 

find ./ -name *.pidb | xargs svn delete --keep-local

Wednesday, December 7, 2011

Cheat Sheet

Below is my linux terminal cheat sheet that I use often. I decided to post it mostly so I could easily find it myself. I seem to reference my own blog quite often to remember how I did something in the past. These are all just examples not command descriptions. I hope it is helpful...

Users, Groups, & Permissions
  • Create User Account
    • sudo adduser <username> # program that does everything for you
  • Modify User Account
    • sudo usermod -a -G <group> <username> #append user to a group
  • Delete User Account
    • ?
  • Change Permissions
    • chmod 755 <myfile>
    • chmod -R 755 <mydir> #applied permissions recursively on directory
  • Change Ownership
    • chown root:root /my/dir/path/
Working w/ Files
  • Create Empty Text File
    • >> <filename>
    • touch <filename>
  • Move Files
    • mv <oldfile> <newfile>
  • Rename Files
    • mv <oldfile> <newfile>
  • Delete File
    • rm <filename>
  • Delete Recursively w/ Wildcard
    • find . -type f -name '*.htm' -delete
  • Copy Files
    • cp <existing> <new>
  • Change line terminators
    • flip -ub <textfile>
  • Search results of standard output
    • grep <searchstring>
Working w/ Directories
  • Create Empty Directory
    • mkdir
  • Move or Rename
    • mv
  • Delete Directory Recursively
    • rm -R <folder> <newfolder>
  • Copy Recursively
    • cp -r <dir1> <dir2>
    • scp ...
    • rsync
Misc. Tools
  • Search Results of Standard Output:
    • grep <searchstring>
System Resources
  • View Memory Usage
    • free
  • Process Monitor
    • top
    • htop
  • Processes
    • ps -A
  • System Activity
    • vmstat
  • User Monitor
    • who
    • whoami
  • View Device Ids (UUID)
    • blkid
Package Management

  • Install
    • apt-get install <packagename>
  • Remove
    • apt-get remove <packagename>
  • Remove completely
    • apt-get remove --purge <packagename>
  • List all installed package
    • dpkg -l
  • See if a package is installed
    • dpkg -l | grep <packagename>
  • Find sources here:
    • sudo nano /etc/apt/sources.list
Networking
  • Find computers on network:
    • arp -av
Automation
  • Schedule tasks:
    • crontab -e
Tar and Zip Files & Directories
  • Tar Files
    • tar -cvpf <file.tar> <files or folder>
  • Tar Files and Zip
    • tar -zcvpf <file.tar.gz> <files or folder>
  • View  Tar
    • tar -ztvf <file.tar>
  • Untar Files
    • tar -xvpf <file.tar> <files or folder>
  • Untar Files w/ Zip
    • tar -xvpzf <file.tar.gz>

Links:

http://www.commandlinefu.com/commands/view/7159/recursive-remove-all-htm-files

Thursday, August 11, 2011

Samba

So I setup a Samba server running on an Ubuntu headless machine. From the terminal go ahead and install the server portion. You could also install smbclient if you need to.
sudo apt-get install samba
Edit the /etc/samba/smb.conf file in your favorite text editor. Make sure the following two lines are not commented out and have the right values for your network.
 workgroup = MyNetwork
 security = user
Now your linux machine will be visible on the windows workgroup you specified and logging on will be secured by a unix account.

You will need to create linux user accounts on the machine for any user you want to use a samba share under this configuration. Also, you will need to add the users to the samba password database. I assume you can figure out how to create user account and add them to the appropriate groups, but this is how you add and enable those users to the samba password file...
sudo smbpasswd -a username
sudo smbpasswd -e username
This allows you to have a lot of user accounts on the computer but control access using samba instead of the OS only.

Now Create a new share at the bottom of the file, or uncomment any of the examples. Here is an example folder share from the Ubuntu server manual...
[share]
    comment = Ubuntu File Server Share
    path = /srv/samba/share
    browsable = yes
    guest ok = yes
    read only = no
    create mask = 0755
comment: a short description of the share. Adjust to fit your needs.

path: the path to the directory to share.

This example uses /srv/samba/sharename because, according to the Filesystem Hierarchy Standard (FHS), /srv is where site-specific data should be served. Technically Samba shares can be placed anywhere on the filesystem as long as the permissions are correct, but adhering to standards is recommended.

browsable: enables Windows clients to browse the shared directory using Windows Explorer.

guest ok: allows clients to connect to the share without supplying a password.

read only: determines if the share is read only or if write privileges are granted. Write privileges are allowed only when the value is no, as is seen in this example. If the value is yes, then access to the share is read only.

create mask: determines the permissions new files will have when created.

Make sure to create the directory that corresponds to this share and set the proper ownership and permissions...
sudo mkdir -p /srv/samba/share
sudo chown user.group /srv/samba/share/
I wanted a couple of different types of shares, so I create the following shares  for my little file server...
[Share]
  comment = "Shared by all users on this system that are part of the sambashare group"
  path = /path/to/share
  browsable = yes
  guest ok = no
  read only = no
  create mask = 0770
  directory mask = 0770
  force group = @sambashare
  valid users = @sambashare

[Rick]
  comment = "Only accessable by the user named Rick"
  path = /path/to/Rick
  browsable = yes
  guest ok = no
  read only = no
  create mask = 0770
  directory mask = 0770
  force user = rick
  force group = rick
  valid users = rick
This is a nice way to have a shared directory that all valid users on the system can access and a private share for each user on the system. I have no idea if this is a standard way of accomplishing this, but it works really well for us.

The "create mask" and "directory mask" options are used to define the default permissions applied to any new files or directories that the user creates. Likewise, the "force user" and "force group" options determine the ownership of the files and directories. Notice for the first share I left out the "force user" option and set the "force group" option to @sambashare. This is nice because the ownership of any new files or directories are owned by the user that created them.

You can restart the two samba services like so to get started...
sudo restart smbd
sudo restart nmbd
Also, the following command is useful to parse the configuration file and see if there were any problems are suggestions...
testparm /etc/samba/smb.conf
Links:

https://help.ubuntu.com/11.10/serverguide/C/samba-fileserver.html
https://help.ubuntu.com/community/Samba/SambaServerGuide
http://www.handson.ca/tutorials/samba.html
http://www.linuxquestions.org/questions/linux-networking-3/samba-cups-access-denied-unable-to-connect-error-322874/

Monday, August 1, 2011

Raid 1

Wow!! I was doing some research into setting up a linux file and print server and came across the concept of software Raid in linux. Are you kidding me!! Raid 0, 1, 5, etc. can all be done at the linux kernal level without the need of any special Raid hardware. The linux community is amazing! I love it.

So, I have an older desktop I wanted to use as a file and print server for the household and decided to set it up with a simple 2 disk Raid 1. This means the 2 hard drives are mirrored. You can have one hard drive completely fail and lose no data at all. This is perfect for a little light duty file server.  This is a bit slower than a one drive system, but that is not important because I am mostly concerned that I don't lose any data from the past 20 years.

I started by connecting 2 identical 300 gigabyte IDE hard drives that I already had in a ~8 year old computer. An old computer should work fine for this sort of thing...no need to spend any money at all.

You can follow the Ubuntu website documentation for setting up the Raid at installation time like I did at...

https://help.ubuntu.com/community/Installation/SoftwareRAID

I setup identical boot and swap partitions on these two identical hard drives, and then created two Raid devices to mirror the respective partitions (md0 & md1). It is quite amazing that you can do all of this at installation time and the drives just start to sink up seamlessly in the background.

Drive 1:
sda1 - /boot
sda5 - /swap

Drive 2:
sdb1 - /boot
sdb5 - /swap

Once you have the Software raid setup, you can use the mdadm software to manage it.

cat /proc/mdstat



sudo mdadm --query --detail /dev/md0



Links:
http://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array
https://help.ubuntu.com/community/Installation/SoftwareRAID
http://en.wikipedia.org/wiki/RAID
https://help.ubuntu.com/10.04/serverguide/C/advanced-installation.html
http://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array
http://nwlinux.com/diagnosing-a-degraded-ubuntu-raid-1-array/
http://advosys.ca/viewpoints/2007/04/setting-up-software-raid-in-ubuntu-server/
http://tldp.org/HOWTO/Software-RAID-HOWTO-2.html
http://ubuntuforums.org/showthread.php?t=408461
http://www.linuxjournal.com/article/6412
http://www.linuxjournal.com/article/5898
http://shop.oreilly.com/product/9781565927308.do
http://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array
http://www.brighthub.com/computing/linux/articles/37236.aspx#
http://tldp.org/HOWTO/Partition/setting_up_swap.html
http://en.wikipedia.org/wiki/Mdadm#Types_of_physical_device
http://www.ducea.com/2009/03/08/mdadm-cheat-sheet/
https://help.ubuntu.com/community/Installation/SoftwareRAID
http://www.handson.ca/tutorials/samba.html

Friday, July 1, 2011

GIS Desktop


This blog started out being mostly about building a Linux GIS server, but I am expanding it now to building a good GIS desktop as well. Below I have an outline of what I hope to cover. As the blog posts are written , I will come back here and add links to those posts. So, these links will either be to my blog posts are external websites.

Great Desktop GIS systems (that run on linux)
Desktop GIS on Ubuntu linux...
Virtual Desktops /w GIS desktop software
Desktop GIS / C# / Mono Software development topics...
Related Development technologies...

Wednesday, June 1, 2011

VirtualBox Daemon

This is a quick post about running VirtualBox guest images as daemons from a headless Ubuntu server host. Most of this can be found in Brendan Kidwell's post located at...

http://www.glump.net/howto/virtualbox_as_a_service

...so I am mostly re-posting it here for myself and others in case his post is ever removed.

First, take a look at the VirtualBox service manager script that is installed with VBox. It is located at /etc/default/virtualbox. This script is run when the computer is shutdown, so it is a good place to tell the computer what to do about an any running virtualbox guest VMs. I modified or added the following lines so it would look for guest images running at my user and save their state, as apposed to just shutting them down.

SHUTDOWN_USERS="user1 user2" # space-delimited list of users who might have runnings vms
SHUTDOWN=savestate # if any are found, suspend them to disk. Could be shutoff too


Next, you need to create a new service manager script to run when the computer starts up. This is a standard startup script in most distributions of linux and usually placed in the /etc/init.d/ directory. You can use the script from the bottom of this post to startup one VirtualBox guest VM, so name it with the guest name and copy it like so...
sudo cp virtualbox-machinename /etc/init.d/virtualbox-machinename

#! /bin/sh
### BEGIN INIT INFO
# Provides:          virtualbox-!!SHORTNAME
# Required-Start:    $local_fs $remote_fs vboxdrv vboxnet
# Required-Stop:     $local_fs $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: !!LONGNAME virtual machine
# Description:       !!LONGNAME virtual machine hosted by VirtualBox
### END INIT INFO
 
# Author: Brendan Kidwell 
#
# Based on /etc/init.d/skeleton from Ubuntu 8.04. Updated for Ubuntu 9.10.
# If you are using Ubuntu <9.10, you might need to change "Default-Stop"
# above to "S 0 1 6".
 
# Do NOT "set -e"
 
# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/usr/sbin:/usr/bin:/sbin:/bin
DESC="!!LONGNAME virtual machine"
NAME=virtualbox-!!SHORTNAME
SCRIPTNAME=/etc/init.d/$NAME
 
MANAGE_CMD=VBoxManage
VM_OWNER=!!USERNAME
VM_NAME="!!LONGNAME" #This has to be the name exactly as it appears in your VirtualBox GUI control panel.
 
# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
 
# Load the VERBOSE setting and other rcS variables
[ -f /etc/default/rcS ] && . /etc/default/rcS
 
# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions
 
#
# Function that starts the daemon/service
#
do_start()
{
 # Return
 #   0 if daemon has been started
 #   1 if daemon was already running
 #   2 if daemon could not be started
 
 sudo -H -u $VM_OWNER $MANAGE_CMD showvminfo "$VM_NAME"|grep "^State:\s*running" >/dev/null && {
     echo "$VM_NAME" is already running.
     return 1
 }
 
 sudo -H -u $VM_OWNER $MANAGE_CMD startvm "$VM_NAME" -type vrdp >/dev/null || {
     echo Failed to start "$VM_NAME".
     return 2
 }
 
 echo "$VM_NAME" started or resumed.
 return 0
}
 
#
# Function that stops the daemon/service
#
do_stop()
{
 # Return
 #   0 if daemon has been stopped
 #   1 if daemon was already stopped
 #   2 if daemon could not be stopped
 #   other if a failure occurred
 
 sudo -H -u $VM_OWNER $MANAGE_CMD showvminfo "$VM_NAME"|grep "^State:\s*running" >/dev/null || {
     echo "$VM_NAME" is already stopped.
     return 1
 }
 
 sudo -H -u $VM_OWNER $MANAGE_CMD controlvm "$VM_NAME" savestate || {
     echo Failed to stop "$VM_NAME".
     return 2
 }
 
 echo "$VM_NAME" suspended.
 return 0
}
 
#
# Display "State" field from showinfo action
#
do_status()
{
 sudo -H -u $VM_OWNER $MANAGE_CMD showvminfo "$VM_NAME"|grep "^State:\s*.*$"
}
 
case "$1" in
  start)
 [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
 do_start
 case "$?" in
  0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
  2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
 esac
 ;;
  stop)
 [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
 do_stop
 case "$?" in
  0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
  2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
 esac
 ;;
  restart|force-reload)
 #
 # If the "reload" option is implemented then remove the
 # 'force-reload' alias
 #
 log_daemon_msg "Restarting $DESC" "$NAME"
 do_stop
 case "$?" in
   0|1)
  do_start
  case "$?" in
   0) log_end_msg 0 ;;
   1) log_end_msg 1 ;; # Old process is still running
   *) log_end_msg 1 ;; # Failed to start
  esac
  ;;
   *)
    # Failed to stop
  log_end_msg 1
  ;;
 esac
 ;;
  status)
 do_status
 ;;
  *)
 #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
 echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload|status}" >&2
 exit 3
 ;;
esac

Then set the permissions so it can be executable and leave the owner set to root

sudo chmod 755 virtualbox-machinename


Now you can test it with...

./etc/init.d/virtualbox-machinename status # Is the VM running?
./etc/init.d/virtualbox-machinename start  # Start the VM
./etc/inid.d/virtualbox-machinename stop   # Stop the VM


Run the command top to see if a VBoxHeadless process is running. Now we need to install the script with the host computer. Use the update-rc.d command to install the script. I like to add another argument (99) to this to make sure that the script is made to run late in the bootup process. That way there is not a problem if this process relies on some other service that has not been run yet.

sudo update-rc.d -f virtualbox-machinename defaults 99


You are done! You can repeat this process for any other virtual machines you would like to run on this host computer.

Links:
https://help.ubuntu.com/community/UbuntuBootupHowto
http://www.glump.net/howto/virtualbox_as_a_service
http://www.liberiangeek.net/2010/05/how-to-start-stop-services-in-ubuntu-lucid-automatically/
http://www.virtualbox.org/manual/ch08.html
http://www.howtoforge.com/vboxheadless-running-virtual-machines-with-virtualbox-4.0-on-a-headless-ubuntu-11.04-server

Sunday, April 24, 2011

Subversion

How about we install and configure a version control system while we are at it. Let's install Subversion...my favorite.
$ sudo apt-get install subversion libapache2-svn
Make the directory that we are going to create the repository in, then create the repository, and finally set the ownership to the user that Apache runs as...
$ sudo mkdir /home/<user>/svn
$ svnadmin create /home/<user>/svn
$ sudo chown -R www-data /home/<user>/svn
Now let's create the password file for accessing the repository and one account. It will ask you to enter a password for the new account twice...
$ sudo htpasswd -c /etc/apache2/dav_svn.passwd <user>
Then let's create an authorization file for giving fine grain read/write permissions to particular directories and files in the repository itself. I am just going to give read/write permissions to the entire repository structure...
$ sudo nano /etc/apache2/dav_svn.authz
...
[/]*= 
<user>=rw
We need to configure it all together now so you can do this in any place you like but the easiest is in the file below or virtual host file...
$ sudo nano /etc/apache2/mods-enabled/dav_svn.conf
Add or uncomment this text...
<Location /svn>
  DAV svn
  SVNPath /home/<user>/svn
  AuthType Basic
  AuthName "Subversion Repository"
  AuthUserFile  /etc/apache2/dav_svn.passwd
  AuthzSVNAccessFile /etc/apache2/dav_svn.authz
  Require valid-user
  SSLRequireSSL
</Location>
Let's grab our old repository from another instance of subversion and bring it over...
$ svnadmin dump /path/to/repository > repo_name.svn_dump
Then lets suck it up into our new repository on this new server...
$ svnadmin load /home/<user>/svn < repo_name.svn_dump
How about adding SSL to make it all secure. Load this module so we can use SSL...
$ sudo a2enmod ssl
$ sudo a2ensite default-ssl
You may also need to install a couple of additional mods to make it all work depending on your linux distribution or version
sudo a2enmod authz_svn
sudo a2enmod dav_svn
Now you should be able to open you subversion repository using https://localhost/svn and it should work. Try using curl or something like it to test it out locally on your headless server like...
$ sudo apt-get install curl
$ curl -k https://username:password@localhost/svn