Amazon

Wednesday, September 14, 2016

JMS - General and Important Concepts

Transacted Message in JMS

In a Java SE environment or in the Java EE application client container:
If transacted is set to true then the session will use a local transaction which may subsequently be committed or rolled back by calling the session's commit or rollback methods. The argument acknowledgeMode is ignored.
·         If transacted is set to false then the session will be non-transacted. In this case the argument acknowledgeMode is used to specify how messages received by this session will be acknowledged. The permitted values are Session.CLIENT_ACKNOWLEDGE, Session.AUTO_ACKNOWLEDGE and Session.DUPS_OK_ACKNOWLEDGE. For a definition of the meaning of these acknowledgement modes see the links below.
Modifier and Type
Field and Description
static int
With this acknowledgment mode, the session automatically acknowledges a client's receipt of a message either when the session has successfully returned from a call to receive or when the message listener the session has called to process the message successfully returns.
static int
With this acknowledgment mode, the client acknowledges a consumed message by calling the message's acknowledge method.
static int
This acknowledgment mode instructs the session to lazily acknowledge the delivery of messages.
static int
This value may be passed as the argument to the method createSession(int sessionMode) on the Connection object to specify that the session should use a local transaction.

In a Java EE web or EJB container, when there is an active JTA transaction in progress:
Both arguments transacted and acknowledgeMode are ignored. The session will participate in the JTA transaction and will be committed or rolled back when that transaction is committed or rolled back, not by calling the session's commit or rollback methods. Since both arguments are ignored, developers are recommended to use createSession(), which has no arguments, instead of this method.

Interface TopicConnectionFactory

Modifier and Type
Method and Description
Creates a topic connection with the default user identity.
createTopicConnection(String userName, String password)
Creates a topic connection with the specified user identity.
public class FirstClient {
      private Context context = null;
      private TopicConnectionFactory factory = null;
      private TopicConnection connection = null;
      private TopicSession session = null;
      private Topic topic = null;
      private TopicPublisher publisher = null;

      public FirstClient() {
            Properties initialProperties = new Properties();
            initialProperties.put(InitialContext.INITIAL_CONTEXT_FACTORY,
                                  "org.exolab.jms.jndi.InitialContextFactory");
            initialProperties.put(InitialContext.PROVIDER_URL, "tcp://localhost:3035");
            try {
                  context = new InitialContext(initialProperties);
                  factory = (TopicConnectionFactory) context.lookup("ConnectionFactory");
                  topic = (Topic) context.lookup("topic1");
                  connection = factory.createTopicConnection();
                 
                  session = connection.createTopicSession(false,  //Transacted
                                                          TopicSession.AUTO_ACKNOWLEDGE);
                  publisher = session.createPublisher(topic);
                  EventMessage eventMessage = new EventMessage(1, "Message from FirstClient");
                  ObjectMessage objectMessage = session.createObjectMessage();
                  objectMessage.setObject(eventMessage);
                  connection.start();
                  publisher.publish(objectMessage);
                  System.out.println(this.getClass().getName()+ " has sent a message : " + eventMessage);
                  session.close();
                  connection.close();
                  context.close();
            } catch (Exception e) {
            }
      }
}

Synchronous Messaging and Asynchronous Messaging

In case of asynchronous consumption, the consumer is implementing the MessageListener interface .So the overridden onMessage() method gets the message. But in case of synchronous consumption, the client is waiting to get the message since we used the receive() method of the consumer.
public interface MessageListener
A MessageListener object is used to receive asynchronously delivered messages.

Each session must ensure that it passes messages serially to the listener. This means that a listener assigned to one or more consumers of the same session can assume that the onMessage method is not called with the next message until the session has completed the last call.
Implement MessageListener Interface for implement Asynchronous Consumer.
Modifier and Type
Method and Description
void
onMessage(Message message)
Passes a message to the listener.


JMS Reliablity Mechanisms-Message persistence in JMS

1)Persistent delivery mode – This is the default  delivery mode. It forces the JMS provider to take extra care to avoid the loss of message in case of failure. Message is persisted in secondary storage. When the provider comes alive again message will be delivered to consumer. This ensures reliable message delivery.
2)Non-Persistent delivery mode– If we specify the delivery mode as non-persistent then the provider is not taking extra effort to persist the message if a provider failure occurs.

By using the setDeliveryMode(int value) method of MessageProducer interface .
DeliveryMode.PERSISTENT =2
DeliveryMode.NON_PERSISTENT=1 .
producer.setDeliveryMode(DeliveryMode.PERSISTENT);

Or Use the overloaded send() or publish() method of MessageProducer interface .The second argument in the method call specifies the delivery mode.
producer.send(message, DeliveryMode.PERSISTENT);

Setting Message Priority Levels in JMS

1.       Priority levels – 0 to 9
2.       Default Priority Level – 4
3.       Messages having highest priority are delivered first.
4.       Setting Message Priority
-By using setPriority(int value) method of MessageProducer interface.            
-By using the overloaded publish() method.The third argument  will be the priority.               
     void send(Message message, int deliveryMode, int priority, long timeToLive) throws JMSException
                topicPublisher.publish(message, DeliveryMode.NON_PERSISTENT, 3,20000)
message - the message to send
deliveryMode - the delivery mode to use
priority - the priority for this message
timeToLive - the message's lifetime (in milliseconds)

Message Expiry in JMS
By default a message never expires.In some cases message will become obsolete after a particular time period. In such situation it is  desirable to set expiration time . After the message expires, it will not be delivered.
We can set the expiration time in program in two ways.
 
Using setTimeToLive() method of MessageProducer interface to set the expiry time of messages from that producer.
Example: topPublisher .setTimeToLive(10000)
The above statement sets the expiry time 10000 milliseconds(10 seconds)
 
Using the overridden method publish() of MessageProducer
Example :topPublisher.publish(message, DeliveryMode.PERSISTENT, 3,10000)
The fourth argument gives the expiry time as 10000 milliseconds
The reliability mechanisms we discussed so far are :
 
Creating Durable Subscriptions in JMS

This ensures message delivery to Subscriber, when it comes alive.
 
topic = (Topic) context.lookup("topic1");
connection = factory.createTopicConnection();
session = connection.createTopicSession(false,TopicSession.AUTO_ACKNOWLEDGE);
subscriber = session.createDurableSubscriber(topic,"sampleSubscription");
// subscriber = session.createSubscriber(topic);
connection.start();
Message message = subscriber.receive();
A durable subscriber can have only one active subscriber at a time.
A  durable subscriber registers a durable subscription with a unique identity (“sampleSubscription” in above example). Subsequent subscribers with the same identity will resume the subscription in the state where the previous subscriber was left.
If there is no active subscriber, then the JMS provider keeps the messages till they are received by any subscriber or till the message expires.
 
 




Monday, May 2, 2016

Hadoop - Practicing HDFS Basic Commands

notroot@ubuntu:~$ hadoop
Usage: hadoop [--config confdir] COMMAND
where COMMAND is one of:
  namenode -format     format the DFS filesystem
  secondarynamenode    run the DFS secondary namenode
  namenode             run the DFS namenode
  datanode             run a DFS datanode
  dfsadmin             run a DFS admin client
  mradmin              run a Map-Reduce admin client
  fsck                 run a DFS filesystem checking utility
  fs                   run a generic filesystem user client
  balancer             run a cluster balancing utility
  fetchdt              fetch a delegation token from the NameNode
  jobtracker           run the MapReduce job Tracker node
  pipes                run a Pipes job
  tasktracker          run a MapReduce task Tracker node
  historyserver        run job history servers as a standalone daemon
  job                  manipulate MapReduce jobs
  queue                get information regarding JobQueues
  version              print the version
  jar            run a jar file
  distcp copy file or directories recursively
  archive -archiveName NAME -p * create a hadoop archi                                                                                        ve
  classpath            prints the class path needed to get the
                       Hadoop jar and the required libraries
  daemonlog            get/set the log level for each daemon
 or
  CLASSNAME            run the class named CLASSNAME
Most commands print help when invoked w/o parameters.

balancer - Runs a cluster balancing utility. An administrator can simply press Ctrl-C to stop the rebalancing process. See Balancer for more details.

notroot@ubuntu:~$ hadoop balancer
16/05/03 01:07:11 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 0 time(s).
16/05/03 01:07:12 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 1 time(s).
16/05/03 01:07:13 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 2 time(s).
16/05/03 01:07:14 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 3 time(s).
16/05/03 01:07:15 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 4 time(s).
16/05/03 01:07:16 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 5 time(s).
16/05/03 01:07:17 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 6 time(s).
16/05/03 01:07:18 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 7 time(s).
16/05/03 01:07:19 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 8 time(s).
16/05/03 01:07:20 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 9 time(s).
Received an IO exception: Call to localhost/127.0.0.1:8020 failed on connection                                                                                         exception: java.net.ConnectException: Connection refused . Exiting...
Balancing took 11.778 seconds

notroot@ubuntu:~$ hadoop version
Hadoop 1.0.3
Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r                                                                                         1335192
Compiled by hortonfo on Tue May  8 20:31:25 UTC 2012
From source with checksum e6b0c1e23dcf76907c5fecb4b832f3be

notroot@ubuntu:~$ hadoop fs -ls hdfs:/
16/05/03 01:08:44 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 0 time(s).
16/05/03 01:08:45 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 1 time(s).
16/05/03 01:08:46 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 2 time(s).
16/05/03 01:08:47 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 3 time(s).
16/05/03 01:08:48 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 4 time(s).
16/05/03 01:08:49 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 5 time(s).
16/05/03 01:08:50 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 6 time(s).
16/05/03 01:08:51 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 7 time(s).
16/05/03 01:08:52 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 8 time(s).
16/05/03 01:08:53 INFO ipc.Client: Retrying connect to server: localhost/127.0.0                                                                                        .1:8020. Already tried 9 time(s).
Bad connection to FS. command aborted. exception: Call to localhost/127.0.0.1:80                                                                                        20 failed on connection exception: java.net.ConnectException: Connection refused

notroot@ubuntu:~$ start-all.sh
starting namenode, logging to /home/notroot/lab/software/hadoop-1.0.3/libexec/..                                                                                        /logs/hadoop-notroot-namenode-ubuntu.out
localhost: starting datanode, logging to /home/notroot/lab/software/hadoop-1.0.3                                                                                        /libexec/../logs/hadoop-notroot-datanode-ubuntu.out
localhost: starting secondarynamenode, logging to /home/notroot/lab/software/had                                                                                        oop-1.0.3/libexec/../logs/hadoop-notroot-secondarynamenode-ubuntu.out
starting jobtracker, logging to /home/notroot/lab/software/hadoop-1.0.3/libexec/                                                                                        ../logs/hadoop-notroot-jobtracker-ubuntu.out
localhost: starting tasktracker, logging to /home/notroot/lab/software/hadoop-1.                                                                                        0.3/libexec/../logs/hadoop-notroot-tasktracker-ubuntu.out

notroot@ubuntu:~$ jps
3248 DataNode
3016 NameNode
3479 SecondaryNameNode
3910 Jps
3804 TaskTracker
3560 JobTracker

notroot@ubuntu:~$ hadoop fs -ls hdfs:/
Found 1 items
drwxr-xr-x   - notroot supergroup          0 2016-05-01 07:24 /home


notroot@ubuntu:~$ hadoop  fsck - /
FSCK started by notroot from /127.0.0.1 for path / at Tue May 03 01:22:23 UTC 20                                                                                        16
.Status: HEALTHY
 Total size:    4 B
 Total dirs:    6
 Total files:   1
 Total blocks (validated):      1 (avg. block size 4 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    1
 Average block replication:     1.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          1
 Number of racks:               1
FSCK ended at Tue May 03 01:22:23 UTC 2016 in 10 milliseconds


The filesystem under path '/' is HEALTHY

notroot@ubuntu:~$ hadoop version
Hadoop 1.0.3
Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1335192
Compiled by hortonfo on Tue May  8 20:31:25 UTC 2012
From source with checksum e6b0c1e23dcf76907c5fecb4b832f3be

notroot@ubuntu:~$ hadoop balancer
Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
16/05/03 01:23:35 INFO net.NetworkTopology: Adding a new node: /default-rack/127.0.0.1:50010
16/05/03 01:23:35 INFO balancer.Balancer: 0 over utilized nodes:
16/05/03 01:23:35 INFO balancer.Balancer: 1 under utilized nodes:  127.0.0.1:50010
The cluster is balanced. Exiting...
Balancing took 2.405 seconds

notroot@ubuntu:~$ hadoop balancer
Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
16/05/03 01:23:50 INFO net.NetworkTopology: Adding a new node: /default-rack/127.0.0.1:50010
16/05/03 01:23:50 INFO balancer.Balancer: 0 over utilized nodes:
16/05/03 01:23:50 INFO balancer.Balancer: 1 under utilized nodes:  127.0.0.1:50010
The cluster is balanced. Exiting...
Balancing took 2.32 seconds

notroot@ubuntu:~$ hadoop fs -ls /
Found 2 items
drwxr-xr-x   - notroot supergroup          0 2016-05-01 07:24 /home
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:23 /system

notroot@ubuntu:~$ hadoop fs -ls /default-rack
ls: Cannot access /default-rack: No such file or directory.


notroot@ubuntu:~$ hadoop fs -ls  /default-rack/127.0.0.1:50010
ls: java.net.URISyntaxException: Relative path in absolute URI: 127.0.0.1:50010
Usage: java FsShell [-ls ]

notroot@ubuntu:~$ hadoop fs -ls hdfs:/
Found 2 items
drwxr-xr-x   - notroot supergroup          0 2016-05-01 07:24 /home
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:23 /system

notroot@ubuntu:~$ hadoop fs -count hdfs:/
           7            1                  4 hdfs://localhost:8020/

notroot@ubuntu:~$ hadoop fs -ls hdfs:/
Found 2 items
drwxr-xr-x   - notroot supergroup          0 2016-05-01 07:24 /home
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:23 /system

notroot@ubuntu:~$ hadoop fs -ls hdfs:/home
Found 1 items
drwxr-xr-x   - notroot supergroup          0 2016-05-01 07:24 /home/notroot

notroot@ubuntu:~$ hadoop fs -ls hdfs:/home/notroot
Found 1 items
drwxr-xr-x   - notroot supergroup          0 2016-05-01 07:24 /home/notroot/lab

notroot@ubuntu:~$ hadoop fs -ls hdfs:/home/notroot/lab
Found 1 items
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:22 /home/notroot/lab/mapred

notroot@ubuntu:~$ hadoop fs -ls hdfs:/home/notroot/lab/mapred
Found 1 items
drwx------   - notroot supergroup          0 2016-05-03 01:22 /home/notroot/lab/mapred/system

notroot@ubuntu:~$ hadoop fs -ls hdfs:/home/notroot/lab/mapred/system
Found 1 items
-rw-------   1 notroot supergroup          4 2016-05-03 01:22 /home/notroot/lab/mapred/system/jobtracker.info

notroot@ubuntu:~$ hadoop fs -ls hdfs:/home/notroot/lab/mapred
Found 1 items
drwx------   - notroot supergroup          0 2016-05-03 01:22 /home/notroot/lab/mapred/system

notroot@ubuntu:~$ hadoop fs -ls hdfs:/
Found 2 items
drwxr-xr-x   - notroot supergroup          0 2016-05-01 07:24 /home
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:23 /system

notroot@ubuntu:~$ hadoop fs -ls hdfs:/system

notroot@ubuntu:~$ ls -lt
total 7608
drwxrwxr-x 2 notroot notroot    4096 May  1 11:14 sanjeev
-rw-rw-r-- 1 notroot notroot 7772980 Apr 26 18:07 latest.tar.gz
drwxrwxr-x 3 notroot notroot    4096 Nov 12  2012 downloads
drwxrwxr-x 7 notroot notroot    4096 Sep 26  2012 lab
drwxrwxr-x 4 notroot notroot    4096 Sep 26  2012 backup

notroot@ubuntu:~$ cd sanjeev/

notroot@ubuntu:~/sanjeev$ ls
sample  sample1

notroot@ubuntu:~/sanjeev$ cd ..

notroot@ubuntu:~$ ls -h -lrt
total 7.5M
drwxrwxr-x 4 notroot notroot 4.0K Sep 26  2012 backup
drwxrwxr-x 7 notroot notroot 4.0K Sep 26  2012 lab
drwxrwxr-x 3 notroot notroot 4.0K Nov 12  2012 downloads
-rw-rw-r-- 1 notroot notroot 7.5M Apr 26 18:07 latest.tar.gz
drwxrwxr-x 2 notroot notroot 4.0K May  1 11:14 sanjeev

notroot@ubuntu:~$ rm latest.tar.gz

notroot@ubuntu:~$ cd lab

notroot@ubuntu:~/lab$ ls
data  hdfs  mapred  programs  software

notroot@ubuntu:~/lab$ cd data/

notroot@ubuntu:~/lab/data$ ls -lrt
total 174832
-rw-rw-r-- 1 notroot notroot 81468050 Jan 10  2012 weblogs
-rw-rw-r-- 1 notroot notroot 88328787 May 25  2012 txns
-rw-rw-r-- 1 notroot notroot   391355 Jun  9  2012 custs
-rw-rw-r-- 1 notroot notroot  8828133 Jun 10  2012 txntab
drwxrwxr-x 2 notroot notroot     4096 Sep  5  2012 images

notroot@ubuntu:~/lab/data$ cd ..

notroot@ubuntu:~/lab$ ls
data  hdfs  mapred  programs  software

notroot@ubuntu:~/lab$ cd ..

notroot@ubuntu:~$ pwd
/home/notroot

notroot@ubuntu:~$ cd sanjeev

notroot@ubuntu:~/sanjeev$ ls
googleroundtable.mp4  sample  sample1

notroot@ubuntu:~/sanjeev$ ls -lt
total 122812
-rw-rw-r-- 1 notroot notroot       175 May  1 11:15 sample1
-rw-rw-r-- 1 notroot notroot       136 May  1 11:09 sample
-rw-rw-r-- 1 notroot notroot 125744560 Apr 14  2014 googleroundtable.mp4

notroot@ubuntu:~/sanjeev$ ls -lt -h
total 120M
-rw-rw-r-- 1 notroot notroot  175 May  1 11:15 sample1
-rw-rw-r-- 1 notroot notroot  136 May  1 11:09 sample
-rw-rw-r-- 1 notroot notroot 120M Apr 14  2014 googleroundtable.mp4

notroot@ubuntu:~/sanjeev$ hadoop dfs -put googleroundtable.mp4 hdfs:/

notroot@ubuntu:~/sanjeev$ hadoop dfs -ls hdfs:/
Found 3 items
-rw-r--r--   1 notroot supergroup  125744560 2016-05-03 01:37 /googleroundtable.mp4
drwxr-xr-x   - notroot supergroup          0 2016-05-01 07:24 /home
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:23 /system

notroot@ubuntu:~/sanjeev$ hadoop dfs -ls hdfs:/home
Found 1 items
drwxr-xr-x   - notroot supergroup          0 2016-05-01 07:24 /home/notroot

notroot@ubuntu:~/sanjeev$ hadoop dfs -mkdir hdfs:/home/sanjeev

notroot@ubuntu:~/sanjeev$ hadoop dfs -mv hdfs:/googleroundtable.mp4 hdfs:/sanjeev/

notroot@ubuntu:~/sanjeev$ hadoop dfs -ls hdfs:/sanjeev
Found 1 items
-rw-r--r--   1 notroot supergroup  125744560 2016-05-03 01:37 /sanjeev

notroot@ubuntu:~/sanjeev$ hadoop dfs -ls hdfs:/
Found 3 items
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:39 /home
-rw-r--r--   1 notroot supergroup  125744560 2016-05-03 01:37 /sanjeev
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:23 /system

notroot@ubuntu:~/sanjeev$ hadoop dfs -rm hdfs:/sanjeev
Deleted hdfs://localhost:8020/sanjeev

notroot@ubuntu:~/sanjeev$ hadoop dfs -put googleroundtable.mp4 hdfs:/

notroot@ubuntu:~/sanjeev$ hadoop dfs -ls hdfs:/
Found 3 items
-rw-r--r--   1 notroot supergroup  125744560 2016-05-03 01:42 /googleroundtable.mp4
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:39 /home
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:23 /system

notroot@ubuntu:~/sanjeev$ hadoop dfs -mkdir hdfs:/sanjeev

notroot@ubuntu:~/sanjeev$ hadoop dfs -ls hdfs:/
Found 4 items
-rw-r--r--   1 notroot supergroup  125744560 2016-05-03 01:42 /googleroundtable.mp4
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:39 /home
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:42 /sanjeev
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:23 /system

notroot@ubuntu:~/sanjeev$ hadoop dfs -mv hdfs:/googleroundtable.mp4 hdfs:/sanjeev/googleroundtable.mp4

notroot@ubuntu:~/sanjeev$ hadoop dfs -ls hdfs:/
Found 3 items
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:39 /home
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:43 /sanjeev
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:23 /system

notroot@ubuntu:~/sanjeev$ hadoop dfs -ls hdfs:/sanjeev
Found 1 items
-rw-r--r--   1 notroot supergroup  125744560 2016-05-03 01:42 /sanjeev/googleroundtable.mp4

notroot@ubuntu:~/sanjeev$ hadoop dfs -cp hdfs:/sanjeev/googleroundtable.mp4 hdfs:/googlemapred.mp4

notroot@ubuntu:~/sanjeev$ hadoop dfs -ls hdfs:/sanjeev
Found 1 items
-rw-r--r--   1 notroot supergroup  125744560 2016-05-03 01:42 /sanjeev/googleroundtable.mp4

notroot@ubuntu:~/sanjeev$ hadoop dfs -ls hdfs:/
Found 4 items
-rw-r--r--   1 notroot supergroup  125744560 2016-05-03 01:44 /googlemapred.mp4
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:39 /home
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:43 /sanjeev
drwxr-xr-x   - notroot supergroup          0 2016-05-03 01:23 /system

notroot@ubuntu:~/sanjeev$ hadoop dfs -copyToLocal hdfs:/googlemapred.mp4 .

notroot@ubuntu:~/sanjeev$ ls -lt
total 245612
-rw-rw-r-- 1 notroot notroot 125744560 May  3 01:46 googlemapred.mp4
-rw-rw-r-- 1 notroot notroot       175 May  1 11:15 sample1
-rw-rw-r-- 1 notroot notroot       136 May  1 11:09 sample
-rw-rw-r-- 1 notroot notroot 125744560 Apr 14  2014 googleroundtable.mp4

notroot@ubuntu:~/sanjeev$ hadoop fs -copyFromLocal sample hdfs:/sanjeev

notroot@ubuntu:~/sanjeev$ hadoop fs -ls hdfs:/sanjeev
Found 2 items
-rw-r--r--   1 notroot supergroup  125744560 2016-05-03 01:42 /sanjeev/googleroundtable.mp4
-rw-r--r--   1 notroot supergroup        136 2016-05-03 01:53 /sanjeev/sample

notroot@ubuntu:~/sanjeev$ hadoop fs -ls hdfs:/sanjeev/sample
Found 1 items
-rw-r--r--   1 notroot supergroup        136 2016-05-03 01:53 /sanjeev/sample

notroot@ubuntu:~/sanjeev$ hadoop fs -cat hdfs:/sanjeev/sample
this is my test file

i m learing hadoop

its workong on vmware

ubantu is on top of vmaware

this is 2nd day class revision..

Sunday, August 16, 2015

Mongo DB - Introduction

mongo db

  1. no support for joins
  2. no support for transactions 
  3. is schemaless
  4. across multiple collections
  5. data stored in documents 


To Retain Scalability, MongoDB
1. dont support joins,
2. don't support transaction management - Docs are hierarchical, so docs can be accessed automically

scalability and Functionality
High Performance


supports documents in same context having different structure

Mongo DB - non-relational json document store

{'firstname':'Andrew',
 'lastname':'ericson',
 'hobbies':['cycling', 'golf', 'photography']}

mongo shell

use test --- to database test
db.things.save({a:1, b:2}) --save a row

db.things.find() --find all document records of things documents

db.things.save({a:2, b:3, d:4}) --save another row

db.things.find({a:1}) --finds first record

Easy to program because it stored in JSON

JSON - www.json.org
array: list of things [...]  [red,green]
Dictonary : maps {key:val,} {a:b, c:d}

Top level in document has to be distionary
{fruid:[apple, pear, manogoe]}


db.user({'name':'andrew', 'city':'london'})

Changing schema of a document:
var j=db.user.findOne({'name':'andrew'})
j.city='paris'
db.user.save()

allowed size of doc - 16mb

restore command - mongodump mongorestore
connect with shell - mongo

findOne Command

{"name":"andrew"}
{"a":4, "b":5, "c":7}
{a:6, b:7, fruit:["apple", "pear", banana"]}



Connecting MongoDB
MongoShell<->MongoDB

Java Code
->SparkJava
->FreeMarker -MVC
MongoJava Driver ->TCP Connection to MongoD
Port - 80, 8082

MongoShell
start mongo shell
mongo
>use test
switch to db test
>db.things.save({a:1, b:2, c:3})
saved in collection things
>db.things.find()
will return 1 document
>db.things.save({a:3, b:4, c:6, d:200})
>db.things.find()
will return two documents
>db.things.find({a:1})
>db.things.save({a:3, b:4, c:6, d:{k:200}})
>db.things.find.preety()

Mongo DB Operations comparison
Ops - Mongo - SQL
Create - Insert - Insert
Read - Find - Select
Update - Update - Update
Delete - Remove - Delete

$mongo - open mongo shell, connects with db and display version of mongodb
->mongo shell is interactive javascript interpretor

-mongoshell contains doskey, up arrow, down arrow key, left arrow and right arrow key for editing
use tab to autocomplete the token in mongoshell

z={a:1}
z.a will return 1
z["a"] will return 1
difference in above is in z.a, a is not considered literal, i.e.
if i say w="a"
then z[w] will return 1 but not in the case of using dot (.) notation. reason is dot(.) notation look for reference or member variable in an object, but the square [] notation treats object more as a pirce of data or dictionary

BSON - binary JSON - Contains all datatypes of JSON
NumberInt(1)
NumberLong(1)
new Date() -> ISODate("2012-10-21T17:41:51.398Z")

use above constructor syntaxes to create objects of number, date etc.

Inserting documents in MongoDB
doc={"name":"smith", "Age":30}
db.people.insert(doc)
db.people.find()
above row will be returned.
All documents insterted in mongodb will have an id, "_id" primary key is unique. This value is immutable, it cannot be changed.

findOne()
db.people.findOne() - selects a random document from the collection people.

db.people.findOne({"name":"Jones"})
selects a document having value of name field as "Jones"

second argument can be used to specify which field will be returned from database.
db.people.findOne({"name":"Jones"}, {"name":true, "_id":false})
->{"name":"Jones"}
if we don't say "_id":false, it will be returned from DB as default behavior.

find()
inserting records using a loop

for(i=0; i<1000 db.scores.insert="" essay="" exam="" for="" i="" j="" names="" quiz="" score="Math.round(Math.random()*100});}}</p" student="" type="">
3000 rows will be inserted.

db.scores.find() - will return 3000 records, but page wise, 20 records per page. type "it" for iterating to next page.

getting formatted output - use pretty() function

db.scores.find().pretty()

Querying using selection

using AND operation
db.scores.find({student:19, type:"essay"})
will select the rows having student id 19 and type essay

$gt and $lt opearator
db.scores.find({score:{$gt:95}})
will return all records having score value greater than 95

db.scores.find({score:{$gt:95}, type:"essay"})
will return all records having score value greater than 95 and type is equal to "essay"

putting more constraints on score

db.scores.find({score:{$gt:95, $lte:98})
$lte means less than or equal to
$gt means greater than


inequality comparison
db.people.find({name:{$lt:"D"}});
will return all the names starting from A, B or C.

db.people.find({name:{$lt:"D", $gt:"B"}});
will return all the names starting from B, or  C.

db.people.insert({name:42});
db.people.find({name:{$lt:"D", $gt:"B"}});
will not select the record having name=42.

Mogodb allows to query based on structure of document and type of values in document
For exmaple
{"name":"Smith", "age":30, "profession":"hacker"}
{"name":"Jones", "age":35, "profession":"baker"}
{"name":"Alice"}
{"name":"Bob"}
{"name":42}
find all the records having field profession
db.people.find({profession:{$exists:true}});

below records will be selected
{"name":"Smith", "age":30, "profession":"hacker"}
{"name":"Jones", "age":35, "profession":"baker"}

find all the records having field profession
db.people.find({profession:{$exists:false}});
will select all records not have profession field value

type based selection

for selecting records having value of name field as string
db.people.find({name:{$type:2}})
type:2 is for String as per BSON Specs.

$regex
Find all rows field name having value containing "a"
db.people.find({name:{$regex:"a"}});

Find all rows field name having value containing "e" as last letter
db.people.find({name:{$regex:"e$"}});

Find all rows field name field start with "A"
db.people.find({name:{$regex:"^A"}});

$or
$or operator is used as key and value is a array of conditions
db..people.find({$or:[name:{$regex:"e$"}}, {age:{$exists:true}}]};
find all rows having name ending with "e" or all rows containing age attribute.

$and
db.people.find({$and:[name:{$regex:"e$"}}, {age:{$exists:true}}]};

Querying inside arrays
Arrays can be queried at the top level

db.account.insert({name:"Howard", favouriutes:["pretzels", "beer"]});

db.accounts.find({favourites:"pretzels"});

Which of the following documents would be returned by this query?
db.products.find( { tags : "shiny" } );
{ _id : 42 , name : "Whizzy Wiz-o-matic", tags : [ "awesome", "shiny" , "green" ] }
{ _id : 1040 , name : "Snappy Snap-o-lux", tags : "shiny" }


db.accounts.find().pretty(); -display in a format

{name:"Howard", favourites:["pretzels", "beer"]}
{name:"Irving", favourites:["pretzels", "beer", "cheese"]}

$all - Finding all rows which contain all elements of the given array in filtr, for ex
db.accounts.find({favourites:{$all:[""pretzels", "beer"]}});
will display both above records because both records contain "pretzels" and "beer"

$in - Finding all rows which contain any of the elements of the given array in filter, for ex
db.accounts.find({favourites:{$in:[""pretzels", "beer"]}});


Querying nested docs with dot notation

{name:"richard", email:{work:"richard@10gen.com", personal:"kreuter@example.com"}}

db.users.find({email:{work:"richard@10gen.com", personal:"kreuter@example.com"}})

this will return above record, by byte by byte comparison. but if you change the order of elements, query will not find the document
for example:

db.users.find({email:{personal:"kreuter@example.com", work:"richard@10gen.com"}})

or below will also not work

db.users.find({email:{work:"richard@10gen.com"}})

how to query embeded fields
db.users.find({"email.work":"richard@10gen.com"})

this will return by looking the contents of embeded field email

Cursors--
cur=db.people.find();null;
cur.hasNext() -> Return true
cur.next() -> Return next row

while(cur.hasNext())printJson(cur.next());

impose limit

cur.limit(5);null;

cur.sort({name:-1}); null;->sort in reverse order

cur.sort({name:-1}).limit(3); null; -> will not print top 3


counting results:
db.scores.count({type:"exam"})

return count of rews requrned from aboive query.

replacing data in DB

db.people.update({name:"Smith"}, {name:"Thompson", salary:50000});

First argument {namne:"Smith"} is where clause
Second argument {name:"Thompson", salary:50000} is the new values updating the existing values of fields. It will discard the previous vlaues of the fields . for example

> db.foo.insert({ "_id" : "Texas", "population" : 2500000, "land_locked" : 1 });
WriteResult({ "nInserted" : 1 })
> db.foo.update({_id:"Texas"},{population:30000000})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.foo.find();
{ "_id" : "Texas", "population" : 30000000 }

updating specific field of a collection using $set
> db.foo.find({"_id" : "Poland"});
{ "_id" : "Poland", "population" : 2500000, "land_locked" : 2 }

> db.foo.update({"_id" : "Poland"}, {$set:{"population" : 200000}});
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

> db.foo.find({"_id" : "Poland"});
{ "_id" : "Poland", "population" : 200000, "land_locked" : 2 }

If the field  does not exist, new field with set value  is created, e.g.
> db.foo.update({"_id" : "Poland"}, {$set:{"language" : "Polish"}});
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.foo.find({"_id" : "Poland"});
{ "_id" : "Poland", "population" : 200000, "land_locked" : 2, "language" : "Polish" }

$inc - increasing value of a field
> db.foo.update({"_id" : "Poland"}, {$inc:{"population" : 1}});
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.foo.find({"_id" : "Poland"});
{ "_id" : "Poland", "population" : 200001, "land_locked" : 2, "language" : "Polish" }

update will not create any new row if it does not exist
> db.foo.update({"_id" : "Englad"}, {$inc:{"population" : 1}});
WriteResult({ "nMatched" : 0, "nUpserted" : 0, "nModified" : 0 })

> db.food.find({"_id" : "Englad"});
>
But update will add a new field if it does not exist
> db.foo.insert({ "_id" : "England",  "land_locked" : 2 });
WriteResult({ "nInserted" : 1 })
> db.foo.update({"_id" : "England"}, {$inc:{"population" : 1}});
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

> db.foo.find({"_id" : "England"});
{ "_id" : "England", "land_locked" : 2, "population" : 1 }

$inc operator can only be applied to numeric value. $inc operator can add new field but $set operator cannot add new field.

$unset - remove a field from document
> db.users.find({ "_id" : "myrnarackham"});
{ "_id" : "myrnarackham", "phone" : "301-512-7434", "country" : "RU" }
> db.users.update({ "_id" : "myrnarackham"}, {$unset:{"country":"RU"}});
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.users.find({ "_id" : "myrnarackham"});
{ "_id" : "myrnarackham", "phone" : "301-512-7434" }
> db.users.update({ "_id" : "myrnarackham"}, {$unset:{"phone":"1"}});
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.users.find({ "_id" : "myrnarackham"});
{ "_id" : "myrnarackham" }


Using $push, $pop, $pull, $pushAll, $pullAll, $addToSet 
> db.friends.insert({_id : "Mike", interests : [ "chess", "botany" ] });
WriteResult({ "nInserted" : 1 })
> db.friends.update( { _id : "Mike" }, { $push : { interests : "skydiving" } } );
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
>  db.friends.find()
{ "_id" : "Mike", "interests" : [ "chess", "botany", "skydiving" ] }
> db.friends.update( { _id : "Mike" }, { $pop : { interests : -1 } } );
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.friends.find()
{ "_id" : "Mike", "interests" : [ "botany", "skydiving" ] }
> db.friends.update( { _id : "Mike" }, { $addToSet : { interests : "skydiving" } } );
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 0 })
> db.friends.find()
{ "_id" : "Mike", "interests" : [ "botany", "skydiving" ] }
> db.friends.update( { _id : "Mike" }, { $pushAll: { interests : [ "skydiving" , "skiing" ] } } );
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.friends.find()
{ "_id" : "Mike", "interests" : [ "botany", "skydiving", "skydiving", "skiing" ] }

$upsert- insert a record if it does not exist, or update the existing one. example
After performing the following update on an empty collection
db.foo.update( { username : 'bar' }, { '$set' : { 'interests': [ 'cat' , 'dog' ] } } , { upsert : true } );

{ "_id" : ObjectId("507b78232e8dfde94c149949"), "interests" : [ "cat", "dog" ], "username" : "bar" }

Removing Documents - Deleting documents from Database.

> db.people.insert({"name":"Alice", "age":20, sex:"f"})
WriteResult({ "nInserted" : 1 })
> db.people.insert({"name":"John", "age":20, sex:"m"})
WriteResult({ "nInserted" : 1 })
> db.people.insert({"name":"Ran", "age":20, sex:"f"})
WriteResult({ "nInserted" : 1 })
> db.people.insert({"name":"Mony", "age":25, sex:"f"})
WriteResult({ "nInserted" : 1 })
> db.people.insert({"name":"Ricky", "age":30, sex:"m"})
WriteResult({ "nInserted" : 1 })
> db.people.insert({"name":"Aster Krtor", "age":24, sex:"m"})
WriteResult({ "nInserted" : 1 })

Works like find() command. Receives argument to filter the rows to remove
> db.people.remove({name:"Alice"})
WriteResult({ "nRemoved" : 1 })

If you want to remove all documents of the collection "people"
> db.people.remove({})
WriteResult({ "nRemoved" : 5 })
> db.people.find()


If you want to remove more efficiently, use drop
db.people.drop()

drop() function is different from remove(). Remove updates collection one by one, but drop removes whole collection datafile at ones.

Document removal is an atomic operation and no document is half removed.







Friday, January 16, 2015

Oracle : Grants and checking Grants for a Schema

How would you enable one schema to do SELECT on another schema?

Step 1: Login to Srouce Schema SOURCE_SCHEMA, to provide grant to client schema
GRANT SELECT ON SRC_TAB TO CLIENT_SCHEMA;

Step 2: Login to Client Schema to select data from table present in source schema
SELECT * FROM SOURCE_SCHEMA.SRC_TAB;

Same steps can be used for UPDATE, DELETE and other ddl and dml grants.


How would you find from Client Schema that which tables of source schema are granted to Client Schema?

select * from USER_TAB_PRIVS where owner like 'SOURCE_SCHEMA';


How would you find the which objects (TABLE, FUNCTION, SEQUENCE etc) are present in a schema?

select * from ALL_OBJECTS where OWNER like 'SOURCE_SCHEMA' and OBJECT_TYPE like 'TABLE';

select * from ALL_OBJECTS where OWNER like 'SOURCE_SCHEMA' and OBJECT_TYPE like 'FUNCTION';

Saturday, September 27, 2014

Updating EAR with Environment Specific deployment descriptor using ANT Script

Scenario:

  1. EAR contains a web application.
  2. The EAR need to be deployed in three client environments e.g. Development, UAT and Production
  3. The different environments required filters in web.xml. But the URL of filters are different for different environment. 
  4. So, before deployment, the web.xml of Development should be there in Web Application (WAR) and same applies before deployment in UAT and Production.
Solution:
  1. Write an Ant Script to update the WAR file with respective web.xml and update the EAR with updated WAR File.
Ant Script:


<!--
ant -Dtar=apptar.tar -Dxml=PRO movwebx -f build.xml all

ant -f build.xml all

-->

<project  name="UPDATE_EAR" default="dist" basedir=".">
    
    <property name="upd_ear.dir" value="upd_ear"/>

    <target name="mkchdir">
<echo>CREATING TMP FOLDER</echo>   
        <mkdir dir="upd_ear"/>
    </target>

    <target name="mov" depends="mkchdir">
<echo>MOVING TAR IN TMP FOLDER</echo> 
        <move file="${tar}" todir="${upd_ear.dir}"/>
    </target>

    <target name="tarxvf" depends="mov">
<echo>UN TAR ${tar} IN TMP FOLDER</echo> 
        <unjar src="${upd_ear.dir}/${tar}" dest="${upd_ear.dir}"/>
    </target>

    <target name="earxvf" depends="tarxvf">
<echo>UN JAR appear.ear IN TMP FOLDER</echo> 
        <unjar src="${upd_ear.dir}/appear.ear" dest="${upd_ear.dir}"/>
    </target>

    <target name="warxvf" depends="earxvf">
<echo>UN JAR appwar.war IN TMP FOLDER</echo> 
        <unwar src="${upd_ear.dir}/appwar.war" dest="${upd_ear.dir}"/>
    </target>

    <target name="movwebx" depends="warxvf">
<echo>MOVE web_${xml}.xml to web.xml</echo> 
        <move file="${upd_ear.dir}/WEB-INF/web_${xml}.xml"  tofile="${upd_ear.dir}/WEB-INF/web.xml" />
    </target>
 

   <target name="updwar" depends="movwebx"> 
  <echo>UPDATE web.xml of ${xml} IN appwar.war</echo> 
<zip destfile="${upd_ear.dir}/appwar.war" update="true">
<fileset dir="${upd_ear.dir}"> 
<include name="WEB-INF/web.xml" /> 
</fileset>
</zip>
  </target>

  
   <target name="updear" depends="updwar"> 
      <echo>UPDATING appear.ear WITH appwar.war</echo>   
 <zip destfile="${upd_ear.dir}/appear.ear" update="true">
<fileset dir="${upd_ear.dir}"> 
<include name="appwar.war" /> 
</fileset>
</zip>
   </target>

   <target name="movear" depends="updear">
         <echo>MOVING EAR FILE</echo>   
        <move file="${upd_ear.dir}/appear.ear"  todir="." />
   </target>

   <target name="movtar" depends="movear">
         <echo>MOVING TAR FILE</echo>   
        <move file="${upd_ear.dir}/${tar}"  todir="." />
   </target>

    <target name="delchdir" depends="movtar">
        <echo>DELETING TMP FOLDER</echo>   
        <delete dir="${upd_ear.dir}"/>
   </target>
 
   <target name="updateear" depends="delchdir"/>

</project>

Amazon Best Sellors

TOGAF 9.2 - STUDY [ The Open Group Architecture Framework ] - Chap 01 - Introduction

100 Feet View of TOGAF  What is Enterprise? Collection of Organization that has common set of Goals. Enterprise has People - organized by co...