curl and wget

curl is similar with wget, and the difference is that wget save the web page and curl display the content of the web page on the screen.

For example,

curl ifconfig.me will show your ip address on the screen through the web site ifconfig.me

Read Full Post »

configure a virtual internal network and use a Ubuntu as the gateway to access internet

Problem:
There is an internal network without internet access, and I want each node of the internal network to be able to access internet.

Solution:
Use one node of the internal network as the gateway. On the gateway node, there are two network interface cards: eht0 and eth1. eth0 connects the internal network and eth1 connects internet. Configure the route table on the gateway node to forward packets from eth0 to eth1.

Example:

Set up 9 Ubuntu virtual machine under Virtualbox with the following hostnames and ip addresses:
ubuntu1 192.168.10.1
ubuntu2 192.168.10.2
ubuntu3 192.168.10.3
ubuntu4 192.168.10.4
ubuntu5 192.168.10.5
ubuntu6 192.168.10.6
ubuntu7 192.168.10.7
ubuntu8 192.168.10.8
ubuntu100 192.168.10.100 (used as gateway)

Configure gateway

set up two network interface cards (Adapter1 and Adapter2) on ubuntu100 192.168.10.100:
Adapter1 connects internal network

Adapter2 connects internet

configure gateway route table:

sudo sh set_gateway.sh

sudo iptables-save
————————————————————————-

refer to: http://www.debian-administration.org/articles/23

zhengqiu@ubuntu100:~/Desktop$ more set_gateway.sh
#!/bin/sh

PATH=/usr/sbin:/sbin:/bin:/usr/bin

#
# delete all existing rules.
#
iptables -F
iptables -t nat -F
iptables -t mangle -F
iptables -X

# Always accept loopback traffic
iptables -A INPUT -i lo -j ACCEPT

# Allow established connections, and those not coming from the outside
iptables -A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -m state –state NEW ! -i eth1 -j ACCEPT
iptables -A FORWARD -i eth1 -o eth0 -m state –state ESTABLISHED,RELATED -j ACCEPT

# Allow outgoing connections from the LAN side.
iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT

# Masquerade.
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE

# Don’t forward from the outside to the inside.
iptables -A FORWARD -i eth1 -o eth1 -j REJECT

# Enable routing.
echo 1 > /proc/sys/net/ipv4/ip_forward
————————————————————————–

configure the two network cards on the gateway server 192.168.10.100:

restart network
$sudo /etc/init.d/networking restart

check route table

check network configuration

Configure node

on each node, configure network interface and nameserver like the example on ubuntu5 192.168.10.5 below:

On each node of the internal network, you will be able to access the internet.

Read Full Post »

Linux: difference between su and su –

without the dash (su anatherusername), only change the UID and username of the system

with the dash (su – anotherusername), change the UID and username of the system, and also inherit the user environment.

Read Full Post »

Run a program automatically as a non-privileged user on Ubuntu boot

The post takes mongoDB shard server as an example to demostrate running a program automatically as a non-privileged user on Ubuntu boot. Every time you reboot the system the mondod shard server starts automatically.

sudo mkdir -p /mongodb/data/shard12

sudo mkdir -p /mongodb/logs

#add a user named mongodb

sudo adduser mongodb

sudo chown -R mongodb:mongodb /mongodb

sudo vi /etc/init.d/run_mongod_shardsvr.sh (add the following two lines in this file)
#!/bin/bash
su – mongodb -c “/home/zhengqiu/mongodb-linux-x86_64-2.2.1/bin/mongod -shardsvr -replSet shard1 -port 27017 -dbpath /mongodb/data/shard12 -logpath /mongodb/logs/shard12.log -fork &”

sudo chmod +x sudo vi /etc/init.d/run_mongod_shardsvr.sh

#asign the script to detault runlevel
sudo update-rc.d run_mongod_shardsvr.sh defaults

reboot the system and the output of the script shown on the screen

Read Full Post »

Install mongoDB on fully distributed virtual network using Virtualbox

Under Virtualbox, install 8 Ubuntu with the following host name and ip address and assign them different roles

ubuntu1 192.168.10.1 shard1
ubuntu2 192.168.10.2 shard1
ubuntu3 192.168.10.3 shard1
ubuntu4 192.168.10.4 shard2
ubuntu5 192.168.10.5 shard2
ubuntu6 192.168.10.6 shard2
ubuntu7 192.168.10.7 config server
ubuntu8 192.168.10.8 route server

all ubuntu have the following configuration:
ubuntu100 192.168.10.100 gateway and DNS
network mask 255.255.255.0
domain yourdomain.com
user name: use the same information
download mongodb-linux-x84_64-2.2.1 to your home directory on each virtual machine

start shard1:

on 192.168.10.1

cd ~/mongodb-linux-x84_64-2.2.1/bin

mkdir -p /mongodb/data/shard11

mkdir -p /mongodb/logs

./mongod -shardsvr -replSet shard1 -dbpath=/mongodb/data/shard11 -logpath=/mongodb/logs/shard11.log -port=27017 -fork

on 192.168.10.2

cd ~/mongodb-linux-x84_64-2.2.1/bin

mkdir -p /mongodb/data/shard12

mkdir -p /mongodb/logs

./mongod -shardsvr -replSet shard1 -dbpath=/mongodb/data/shard12 -logpath=/mongodb/logs/shard12.log -port=27017 -fork

on 192.168.10.3

cd ~/mongodb-linux-x84_64-2.2.1/bin

mkdir -p /mongodb/data/shard13

mkdir -p /mongodb/logs

./mongod -shardsvr -replSet shard1 -dbpath=/mongodb/data/shard13 -logpath=/mongodb/logs/shard13.log -port=27017 -fork

Run the command shown in the following figure to confige shard1:

start shard2:

on 192.168.10.4

cd ~/mongodb-linux-x84_64-2.2.1/bin

mkdir -p /mongodb/data/shard21

mkdir -p /mongodb/logs

./mongod -shardsvr -replSet shard2 -dbpath=/mongodb/data/shard21 -logpath=/mongodb/logs/shard21.log -port=27017 -fork

on 192.168.10.5

cd ~/mongodb-linux-x84_64-2.2.1/bin

mkdir -p /mongodb/data/shard22

mkdir -p /mongodb/logs

./mongod -shardsvr -replSet shard2 -dbpath=/mongodb/data/shard22 -logpath=/mongodb/logs/shard22.log -port=27017 -fork

on 192.168.10.6

cd ~/mongodb-linux-x84_64-2.2.1/bin

mkdir -p /mongodb/data/shard23

mkdir -p /mongodb/logs

./mongod -shardsvr -replSet shard2 -dbpath=/mongodb/data/shard23 -logpath=/mongodb/logs/shard23.log -port=27017 -fork

 

On any node of shard1, run the command shown in the following figure to configure shard1:

On any node of shard2, run the command shown in the following figure to configure shard2:

start config server on ubuntu7 (192.168.10.7):

start route server on ubuntu8 (192.168.10.8):

configure shard cluster:
run ./mongo localhost:30000 on the route server ubuntu8 (192.168.10.8), pay attention to the port number 30000, which is the port number where route server listens to

Note: addshard command should “use admin”

check if shard cluster configured successfully:

create a collection named table1 under the database testDB and insert 1000000 documents (records) without enabling sharding. In this case, all documents are located in one shard.

enable sharding on database. This does not automatically shard any collections, but makes it possible to begin sharding collections using sh.shardCollection() or db.run.Command({shardcollections}).

enable sharding on collection (table) and collection table1 got splitted over the two shards. Sharding is on a per-collection basis. We must explicitly specify collections to shard. The collections must belong to a database for which sharding has been enabled.

The following figures show an example of how sharding happens:
create a database named testDB and create a collection named table1 and insert 1000000 documents into this collection. enablesharding the database does not shard table1 illustrated by the first db.stats(). shardcollection does shard table1, which is illustrated by the second and third db.stats(). The second db.stats() still shows table1 not sharded because the sharding process takes some time to finish.

 

——————————————————————————————————————————–

 

How to change chunk size?

refer to http://docs.mongodb.org/manual/administration/sharding/#sharding-balancing-modify-chunk-size

When you initialize a sharded cluster, the default chunk size is 64 megabytes. This default chunk size works well for most deployments. However, if you notice that automatic migrations are incurring a level of I/O that your hardware cannot handle, you may want to reduce the chunk size. For the automatic splits and migrations, a small chunk size leads to more rapid and frequent migrations.

To modify the chunk size, use the following procedure:
1. Connect to any mongos in the cluster using the mongo shell.

2.Issue the following command to switch to the Config Database Contents:

use config

3. Issue the following save() operation:
db.settings.save( { _id:”chunksize”, value: <size> } )

Where the value of <size> reflects the new chunk size in megabytes. Here, you’re essentially writing a document whose values store the global chunk size configuration value.

Note:

The chunkSize and –chunkSize options, passed at runtime to the mongos do not affect the chunk size after you have initialized the cluster.

To eliminate confusion you should always set chunk size using the above procedure and never use the runtime options.

Modifying the chunk size has several limitations:
•Automatic splitting only occurs when inserting documents or updating existing documents.
•If you lower the chunk size it may take time for all chunks to split to the new size.
•Splits cannot be “undone.”

If you increase the chunk size, existing chunks must grow through insertion or updates until they reach the new size.

————————————————————————————-

collection remove() and drop() difference

 

remove a database

show content of a collection

show sharding status

 

test GridFS. On the route server ubuntu8 (192.168.10.8), create a file and upload it to GridFS

Read Full Post »

Ubuntu full screen display problem in virtualbox

If ubuntu does not show in full screen, try:

install virtualbox guest additions: Devices/Install Guest Additions…

reboot the virtual machine

if it is still not working

open a terminal in the virtual ubuntu and install virtualbox guest addition:

$sudo apt-get install virtualbox-ose-guest-x11

reboot the virtual machine

Read Full Post »

Hadoop Pig: ERROR 2999: Unexpected internal error. Failed to create DataStorage

Solution:

First, check apache web site to make sure your Hadoop and Pig versions are compatible.

Then, make sure the following two environment variables are set correctly:

export PIG_CLASSPATH=~/hadoop-0.23.4/etc/hadoop

export HADOOP_HOME=~/hadoop-0.23.4

Read Full Post »