Setting Up Cisco L2 and L3 Devices with GNS3 (1.5.2) for CCNA/CCNP Preparations

See my LinkedIn post on this topic: https://www.linkedin.com/pulse/setting-up-cisco-l2-l3-devices-gns3-152-ccnaccnp-preparations-yee

Install iPython (Jupyter) Notebook on Amazon EMR

1. Use the bootstrap script on this link to install iPython Notebook: https://github.com/awslabs/emr-bootstrap-actions/tree/master/ipython-notebook
2. Although the iPython server is running, it's not integrated with Spark. Follow the instructions according to this blog post: https://districtdatalabs.silvrback.com/getting-started-with-spark-in-python
3. Create the initial SparkContext and SQL context as follows:

from pyspark import  SparkContext
sc = SparkContext( 'local', 'pyspark')

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)

MySQL Driver Error in Apache Spark

I was following the Spark example to load data from MySQL database. See "http://spark.apache.org/examples.html"

There was an error upon executing:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 20.0 failed 4 times, most recent failure: Lost task 0.3 in stage 20.0 (TID 233, ip-172-22-11-249.ap-southeast-1.compute.internal): java.lang.IllegalStateException: Did not find registered driver with class com.mysql.jdbc.Driver

To force Spark to load the "com.mysql.jdbc.Driver", add the following option as highlighted below

val df = sqlContext
  .read
  .format("jdbc")
  .option("url", url)
  .option("dbtable", "people") 
  .option("driver","com.mysql.jdbc.Driver").load()




Install New Interpreter in Zeppelin 0.6.x

In new Zeppelin 0.6.x, you can install new interpreters as follows:

• List all available interpreter: 

1. /usr/lib/zeppelin/bin/install-interpreter.sh --list

• To install the specific interpreters: 

1. /usr/lib/zeppelin/bin/install-interpreter.sh --name jdbc,hbase,postgresql

IAM Errors when Creating Amazon EMR

There are errors related to the lack of permissions in the EMR_EC2_DefaultRole whenever I launch a Amazon EMR cluster. After some searching on the support forum, the default EMR role may not be created automatically for you. Hence, I removed the old default role and created new one as follows:

1. Create default role: 
• aws emr create-default-roles
2. Create instance profile: 
• aws iam create-instance-profile --instance-profile-name EMR_EC2_DefaultRole
3. Verify that instance profile exists but doesn't have any roles:
• aws iam get-instance-profile --instance-profile-name EMR_EC2_DefaultRole
4. Add the role using:
• aws iam add-role-to-instance-profile --instance-profile-name EMR_EC2_DefaultRole --role-name EMR_EC2_DefaultRole

Unstuck Spark/Zeppelin Jobs on Amazon EMR

Apache Zeppelin + Apache Spark is a perfect match. Basically, you can do the following in one console:

• Data Ingestion
• Data Discovery
• Data Analytics
• Data Visualization & Collaboration

As it's still under incubation, the error handling is still not as rock solid. Often, I have experienced Spark jobs being stuck for long time. Usually, restarting the Spark interpreter should do the trick. However, there are times that this simple trick won't work and the only way is to restart the Zeppelin daemon. On Amazon EMR console, do the following:

1. /usr/lib/zeppelin/bin/zeppelin-daemon.sh stop
2. /usr/lib/zeppelin/bin/zeppelin-daemon.sh start

If you wish to execute the scripts in zepplin account, which has a nologin shell. Execute following instead:

1. sudo -s /bin/bash -c '/usr/lib/zeppelin/bin/zeppelin-daemon.sh stop' zeppelin
2. sudo -s /bin/bash -c '/usr/lib/zeppelin/bin/zeppelin-daemon.sh start' zeppelin

If you encounter this Java connection error: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method), it's probably because Zeppelin starts the spark interpreter in a different process.

1. Edit /etc/spark/conf/spark-defaults.conf
2. Comment off the following line and restart Zeppelin

#spark.driver.memory              5g

Reference: http://stackoverflow.com/questions/32735645/hello-world-in-zeppelin-failed

Multiple JSON Configurations for Amazon EMR cluster

To use multiple JSON configurations when you launch the new Amazon EMR cluster, I want to configure Spark to use dynamic allocation of executors and store Zeppelin notebook on S3 storage. Rename the bold red below according to your S3 bucket location. In the following example, create the folder '/user/notebook' under your-s3-bucket. You'll see new note.json under the S3 folder, as you create new Zeppelin notebooks.

[
    {
        "classification":"spark-defaults", 
        "properties": {
            "spark.serializer":"org.apache.spark.serializer.KryoSerializer", 
            "spark.dynamicAllocation.enabled":"true"}, 
        "configurations":[]
    },
    {
        "configurations":[
         {
            "classification":"export",
            "properties":{
               "ZEPPELIN_NOTEBOOK_S3_BUCKET":"your-s3-bucket",
               "ZEPPELIN_NOTEBOOK_STORAGE":"org.apache.zeppelin.notebook.repo.S3NotebookRepo",
               "ZEPPELIN_NOTEBOOK_USER":"user" 
            }
         }
      ],
      "classification":"zeppelin-env",
      "properties":{
      }
   }
]

Run Node.JS within Sublime Text editor

Step 1: Open "Sublime Text 2" editor
Step 2: Tools -> Build System -> New Build System
Step 3: New tab appears. Replace the content with the following lines.

{
"cmd": ["node", "$file", "$file_base_name"],
"working_dir": "${project_path:${folder}}",
"selector": "*.js"
}

Step 4: Save the file and rename it with "NodeJS.sublime-build".
Step 5: Select "Tools -> Build System -> NodeJS"
Step 6: Go to your source program and press "Ctrl-B" to run code.

Connect to WS2012 WSUS Internal Database

To connect to the Windows Internal Database (WID) WSUS, install SQL Management Studio and connect to the server using:

\\.\pipe\MICROSOFT##WID\tsql\query

You may confirm the SQL instance name in red against the startup service description using services.msc.

As there is a constant error of "WSUS server is still processing a previous configuration change", this is what I need to execute on the database instance:

USE SUSDB; 

UPDATE tbSingletonData SET ResetStateMachineNeeded = 0

Develop Apache Spark Apps with IntelliJ IDEA on Windows OS

I've posted an LinkedIn article on "Develop Apache Spark Apps with IntelliJ IDEA on Windows OS".

Making Sense of "User-Based Recommender in 5 minutes"

Have you wondered how Amazon recommend new items to you? This is an  example of Machine Learning implementation, which is a type of Artificial Intelligence. I have followed an introductory example of Apache Mahout and shared this on LinkedIn.

Introducing Apache Pig on Amazon EMR

I've just published "Introducing Apache Pig on Amazon EMR" on LinkedIn.

Develop Hadoop Apps with HortonWorks and IntelliJ IDEA on Windows OS

I have published "Develop Hadoop Apps with HortonWorks and IntelliJ IDEA on Windows OS" on LinkedIn.

Collecting #SEAGames2015 Tweets for Sentiment Analysis on AWS

I have published "Collecting #SEAGames2015 Tweets for Sentiment Analysis on AWS" on LinkedIn.

Getting Started with Data Visualization using Tableau

I have published an article on LinkedIn - "Getting Started with Data Visualization using Tableau"

Getting Started with Amazon Elastic MapReduce for Big Data

I have published an LinkedIn article on "Getting Started with Amazon Elastic MapReduce for Big Data".

Let us overcome the fear & anxiety of Cloud

I have come across this article "Employing Cloud Into 2015" that predicts this year is when cloud would go mainstream. These are the statements that I agree most with:

"Enterprises that embrace cloud computing reduce the amount of IT time and budget devoted to legacy systems and routine upgrades, which then increases the time and budget they have for more innovative projects. When IT innovation happens, business innovation is reached, which then supports job creation." – IDC Chief Research Officer John F. Gantz

“For most organizations, cloud computing should be a no-brainer, given its ability to increase IT innovation and flexibility, lower capital costs, and help generate revenues that are multiples of spending,” said John F. Gantz, chief research officer and senior vice president at IDC.

So, is Y2015 the year of cloud adoption? In fact, cloud has already gone mainstream into our personal lives since many years ago. Look at how cloud companies like Facebook, Google and Amazon change the way we live, play, communicate, buy and sell. Yet, many still remain indifferent in our workplace when come to cloud adoption. It's not about data security or reliability anymore. It has been proven how Amazon and Google run and build data centers with much higher standards than most of ours - be it reliability, security and even cost-efficiency. So what's stopping cloud adoption at our workplace, even though all of us have been living in cloud for such a long time? I believe many of us, especially the younger GenY-ers, probably can't live without it (besides their smartphones).

I suspect it's ALL about fears and job insecurity. For long, we have been reading about how cloud can automate and make things happen faster and cheaper. Unspoken fearful questions might creep in, "Would I still be needed?", "Would I be replaced?", "How could I justify for my next performance review and promotion?". Instead of finding out more about cloud adoption, many simply run away from it (and eventually be left behind).

Yes, cloud would certainly take away mundane and boring jobs today. Tomorrow, it will bring in new exciting innovative jobs that would propel us ahead of (or at least keep up to) our competitors. To smooth this transition that is inevitable, we should play our parts as cloud evangelists to help bring enlightenment to our friends and colleagues. Knowledge is the ultimate anecdote to overcome fear and anxiety. Signing up for free tier services provided by cloud providers, like Amazon Web Services and Microsoft Azure, is the first step toward cloud enlightenment.

Installing OwnCloud on AWS

I came across this blog post that shows how to install OwnCloud on AWS. The post contains several typo errors that should be easy to spot out.

First experience using AWS Virtual Private Cloud

All the while, Amazon Web Services (AWS) - the leading public cloud provider - has been advocating for the demise of On-Premise Private Cloud. Its nearest alternative offering is Virtual Private Cloud (VPC) where you could really build one anytime, anywhere. Using its free usage tier, I've built a base VPC with two EC2 instances (or VMs) as depicted below.

To learn AWS, we have to understand its terminology:

1. What's EC2 Instance? It's Virtual Machine.
2. What's VPC? Virtual network on cloud where you can create multiple IP subnets on it. EC2 instances may be hosted on a VPC.
3. What's Security Group? Think of it like a L2 firewall where you can configure the network access rules e.g. only allow HTTPS to the public Web instance from Internet etc. It is associated to one or more instances.
4. What's Subnet? The usual IP subnet that we knows of. A VPC is made up of one or more subnets. You can configure which subnet is public facing and which are not. In my example, 10.10.1.0/24 is public facing and 10.10.4.0/24 to host my internal instances.
5. What's Network ACL? Think of it like the usual network ACL applied to router interfaces. The ACL is stateless, so you've to define both inbound and outbound for a particular traffic. It can be used to complement the Security Group. For example, allow inbound TCP 443 to the subnet that hosts the above Web instance.
6. What's Elastic IP (EIP)? A public IP assigned to a public facing instance, although only private IP is assigned physically on its NIC. Think of it like an NAT address on the invisible Internet gateway.

To begin with free trial:

1. Of course, create an AWS account using your credit card. Don't worry, AWS won't charge anything to your card, as long as you stay within the free usage tier. You can enable bills monitoring if you're concern that you would exceed the free tier limit. As for me, the ultimate backstopper is to make friend with the extremely friendly AWS account managers.
2. Start with the AWS Quick Start guides, especially the RD Gateway guide.
3. Create a VPC with 2 subnets - one public facing (i.e. RD Gateway for remote admin) and another private subnet to host the internal instances.
4. Launch new instances with a wide range of Amazon Machine Image (AMI) templates to select, including various Windows Server and Linux OSes.
5. Configure the Security Group to allow inbound RDP TCP 3389 for the initial setup of RD Gateway instance.
6. After the RD Gateway is successfully setup, you can tighten network security by allowing only HTTPS traffic.

So far, the usage experience on AWS is good, as though I’m working on my own private cloud. The free SDN feature provided by AWS is also almost as agile and flexible as the VMWare NSX that I've recently experimented with. I’m also impressed by the AWS powershell supports embedded in the Windows template. Most importantly, all the AWS features are well documented. The only ‘complaint’ so far is the relative slow loading of html AWS documentation (probably not hosted/cached in Singapore?)

But can AWS really replace all on-premise private cloud networks? It definitely hold promises due to its great elasticity and flexibility. The next challenge depends on how fast its metering jump, whereas in private cloud world where metering is rarely looked at (lest even use). Much like the debate of whether it's more economical of hiring taxi daily vs owning a car, which cost can be astronomical in Singapore. 

My latest DIY Computer

My latest DIY computer: i7 CPU, 16GB RAM, full SSD drive, Nvidia GTX 650 GPU, Gold-class PSU and a brand new LED monitor. In a blink of eye, Windows and most apps will fire up instantly without delay. All in for just SGD 1,600. Realised my dream to have a home "Data Center in a Box" by enabling Hyper-V for both entertainment and R&D purposes. I can bet that this monster can run faster than all the 5-figure and 6-figure servers at my workplace.

As for Cisco routers simulation, I'll need VMWare ESXi for the CSR1000V. I'll work on ESXi USB stick for alternate boot, using my old laptop to vSphere in.