tag:blogger.com,1999:blog-6410420776811067652024-03-19T16:48:41.446+08:00A Networker's Log FileI have a wide scope of interests in IT, which includes hyper-v private cloud, remote desktop services, server clustering, PKI, network security, routing & switching, enterprise network management, MPLS VPN on enterprise network etc. Started this blog for my quick reference and to share technical knowledge with our team members.samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.comBlogger238125tag:blogger.com,1999:blog-641042077681106765.post-49609994666338183622016-08-27T16:47:00.001+08:002016-08-27T21:51:57.283+08:00Setting Up Cisco L2 and L3 Devices with GNS3 (1.5.2) for CCNA/CCNP PreparationsSee my LinkedIn post on this topic: <a href="https://www.linkedin.com/pulse/setting-up-cisco-l2-l3-devices-gns3-152-ccnaccnp-preparations-yee">https://www.linkedin.com/pulse/setting-up-cisco-l2-l3-devices-gns3-152-ccnaccnp-preparations-yee</a><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-4WmgZ3_k0YA/V8Ga7in5iMI/AAAAAAAAEMA/vpfSpf8pDGwzQBkuJQjpIwjQd8tNIFw3ACLcB/s1600/network.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="425" src="https://4.bp.blogspot.com/-4WmgZ3_k0YA/V8Ga7in5iMI/AAAAAAAAEMA/vpfSpf8pDGwzQBkuJQjpIwjQd8tNIFw3ACLcB/s640/network.jpg" width="640" /></a></div>
<br />samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com58tag:blogger.com,1999:blog-641042077681106765.post-17474784172927186022016-08-15T18:25:00.000+08:002016-08-15T18:25:21.895+08:00Install iPython (Jupyter) Notebook on Amazon EMR<br />
<ol>
<li>Use the bootstrap script on this link to install iPython Notebook: <a href="https://github.com/awslabs/emr-bootstrap-actions/tree/master/ipython-notebook">https://github.com/awslabs/emr-bootstrap-actions/tree/master/ipython-notebook</a></li>
<li>Although the iPython server is running, it's not integrated with Spark. Follow the instructions according to this blog post: <a href="https://districtdatalabs.silvrback.com/getting-started-with-spark-in-python">https://districtdatalabs.silvrback.com/getting-started-with-spark-in-python</a></li>
<li>Create the initial SparkContext and SQL context as follows:</li>
</ol>
<div>
</div>
<br />
<pre style="background-color: #f3f6fa; border-radius: 0.3rem; border: 1px solid rgb(220, 230, 240); box-sizing: border-box; overflow: auto; padding: 0.8rem; word-break: normal; word-wrap: normal;"><span style="color: #567482; font-family: consolas, liberation mono, menlo, courier, monospace;"><span style="font-size: 11.97px; letter-spacing: 0.2px; line-height: 17.3565px;">from pyspark import SparkContext
sc = SparkContext( 'local', 'pyspark')</span></span></pre>
<pre style="background-color: #f3f6fa; border-radius: 0.3rem; border: 1px solid rgb(220, 230, 240); box-sizing: border-box; overflow: auto; padding: 0.8rem; word-break: normal; word-wrap: normal;"><span style="color: #567482; font-family: consolas, liberation mono, menlo, courier, monospace;"><span style="font-size: 11.97px; letter-spacing: 0.2px; line-height: 17.3565px;">from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)</span></span></pre>
samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com10tag:blogger.com,1999:blog-641042077681106765.post-63887640791139862342016-08-12T10:49:00.002+08:002016-08-12T16:31:39.984+08:00MySQL Driver Error in Apache SparkI was following the Spark example to load data from MySQL database. See "http://spark.apache.org/examples.html"<br />
<br />
There was an error upon executing:<br />
<i>org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 20.0 failed 4 times, most recent failure: Lost task 0.3 in stage 20.0 (TID 233, ip-172-22-11-249.ap-southeast-1.compute.internal): java.lang.IllegalStateException: Did not find registered driver with class com.mysql.jdbc.Driver</i><br />
<i><br /></i>
To force Spark to load the "com.mysql.jdbc.Driver", add the following option as highlighted below<br />
<pre style="border-radius: 4px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; color: #333333; font-family: menlo, monaco, consolas, "courier new", monospace; font-size: 13px; line-height: 18.5714px; margin-bottom: 10px; padding: 9.5px; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;"><code class="language-scala" data-lang="scala" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: menlo, "lucida console", consolas, monospace; font-size: inherit; padding: 0px;"><span class="k" style="background-color: transparent; box-sizing: border-box; color: #007020; font-weight: bold;">val</span> <span class="n" style="background-color: transparent; box-sizing: border-box;">df</span> <span class="k" style="background-color: transparent; box-sizing: border-box; color: #007020; font-weight: bold;">=</span> <span class="n" style="background-color: transparent; box-sizing: border-box;">sqlContext</span>
<span class="o" style="background-color: transparent; box-sizing: border-box; color: #666666;">.</span><span class="n" style="background-color: transparent; box-sizing: border-box;">read</span>
<span class="o" style="background-color: transparent; box-sizing: border-box; color: #666666;">.</span><span class="n" style="background-color: transparent; box-sizing: border-box;">format</span><span class="o" style="background-color: transparent; box-sizing: border-box; color: #666666;">(</span><span class="s" style="background-color: transparent; box-sizing: border-box; color: #4070a0;">"jdbc"</span><span class="o" style="background-color: transparent; box-sizing: border-box; color: #666666;">)</span>
<span class="o" style="background-color: transparent; box-sizing: border-box; color: #666666;">.</span><span class="n" style="background-color: transparent; box-sizing: border-box;">option</span><span class="o" style="background-color: transparent; box-sizing: border-box; color: #666666;">(</span><span class="s" style="background-color: transparent; box-sizing: border-box; color: #4070a0;">"url"</span><span class="o" style="background-color: transparent; box-sizing: border-box; color: #666666;">,</span> <span class="n" style="background-color: transparent; box-sizing: border-box;">url</span><span class="o" style="background-color: transparent; box-sizing: border-box; color: #666666;">)</span>
<span class="o" style="background-color: transparent; box-sizing: border-box; color: #666666;">.</span><span class="n" style="background-color: transparent; box-sizing: border-box;">option</span><span class="o" style="background-color: transparent; box-sizing: border-box; color: #666666;">(</span><span class="s" style="background-color: transparent; box-sizing: border-box; color: #4070a0;">"dbtable"</span><span class="o" style="background-color: transparent; box-sizing: border-box; color: #666666;">,</span> <span class="s" style="background-color: transparent; box-sizing: border-box; color: #4070a0;">"people"</span><span class="o" style="background-color: transparent; box-sizing: border-box; color: #666666;">) </span>
<b style="background-color: yellow;">.option("driver","com.mysql.jdbc.Driver")</b><span class="o" style="background-color: transparent; box-sizing: border-box; color: #666666;">.</span><span class="n" style="background-color: transparent; box-sizing: border-box;">load</span><span class="o" style="background-color: transparent; box-sizing: border-box; color: #666666;">()</span>
</code></pre>
<div>
<code class="language-scala" data-lang="scala" style="background: transparent; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: Menlo, "Lucida Console", Consolas, monospace; font-size: inherit; padding: 0px;"><span class="o" style="box-sizing: border-box; color: #666666;"><br /></span></code></div>
samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com9tag:blogger.com,1999:blog-641042077681106765.post-48716271045797888762016-08-10T22:15:00.002+08:002016-08-10T22:25:09.563+08:00Install New Interpreter in Zeppelin 0.6.xIn new Zeppelin 0.6.x, you can <a href="http://zeppelin.apache.org/docs/0.6.0/manual/interpreterinstallation.html" target="_blank">install new interpreters</a> as follows:<br />
<br />
<br />
<ul>
<li>List all available interpreter: </li>
</ul>
<ol>
<li><pre style="background-color: #f3f6fa; border-radius: 0.3rem; border: 1px solid rgb(220, 230, 240); box-sizing: border-box; color: #567482; font-family: Monaco, Menlo, Consolas, "Courier New", monospace; font-size: 12.6px; letter-spacing: 0.2px; line-height: 1.45; overflow: auto; padding: 0.8rem; white-space: pre-wrap; word-break: normal; word-wrap: normal;"><code class="text language-text" data-lang="text" style="background: transparent; border-radius: 0.3rem; border: 0px; box-sizing: border-box; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 11.97px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre; word-break: normal; word-wrap: normal;"><span style="background-color: transparent;">/usr/lib/zeppelin/bin</span>/install-interpreter.sh --list</code></pre>
</li>
</ol>
<ul>
<li>To install the specific interpreters: </li>
</ul>
<ol>
<li><pre style="background-color: #f3f6fa; border-radius: 0.3rem; border: 1px solid rgb(220, 230, 240); box-sizing: border-box; color: #567482; font-family: Monaco, Menlo, Consolas, "Courier New", monospace; font-size: 12.6px; letter-spacing: 0.2px; line-height: 1.45; overflow: auto; padding: 0.8rem; white-space: pre-wrap; word-break: normal; word-wrap: normal;"><code class="text language-text" data-lang="text" style="background: transparent; border-radius: 0.3rem; border: 0px; box-sizing: border-box; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 11.97px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre; word-break: normal; word-wrap: normal;"><span style="background-color: transparent;">/usr/lib/zeppelin/bin</span>/install-interpreter.sh --name jdbc,hbase,postgresql</code></pre>
</li>
</ol>
<div>
<br /></div>
samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com1tag:blogger.com,1999:blog-641042077681106765.post-30088664574596013772016-08-05T15:28:00.001+08:002016-08-05T15:33:57.917+08:00IAM Errors when Creating Amazon EMR There are errors related to the lack of permissions in the EMR_EC2_DefaultRole whenever I launch a Amazon EMR cluster. After some searching on the support forum, the default EMR role may not be created automatically for you. Hence, I removed the old default role and created new one as follows:<br />
<ol>
<li>Create default role: </li>
<ul>
<li>aws emr create-default-roles</li>
</ul>
<li>Create instance profile: </li>
<ul>
<li>aws iam create-instance-profile --instance-profile-name EMR_EC2_DefaultRole</li>
</ul>
<li>Verify that instance profile exists but doesn't have any roles:</li>
<ul>
<li>aws iam get-instance-profile --instance-profile-name EMR_EC2_DefaultRole</li>
</ul>
<li>Add the role using:</li>
<ul>
<li>aws iam add-role-to-instance-profile --instance-profile-name EMR_EC2_DefaultRole --role-name EMR_EC2_DefaultRole</li>
</ul>
</ol>
samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com2tag:blogger.com,1999:blog-641042077681106765.post-84315019823079862922016-07-07T20:27:00.001+08:002016-08-05T11:29:26.152+08:00Unstuck Spark/Zeppelin Jobs on Amazon EMRApache Zeppelin + Apache Spark is a perfect match. Basically, you can do the following in one console:<br />
<br />
<ul>
<li>Data Ingestion</li>
<li>Data Discovery</li>
<li>Data Analytics</li>
<li>Data Visualization & Collaboration</li>
</ul>
<div>
As it's still under incubation, the error handling is still not as rock solid. Often, I have experienced Spark jobs being stuck for long time. Usually, restarting the Spark interpreter should do the trick. However, there are times that this simple trick won't work and the only way is to restart the Zeppelin daemon. On Amazon EMR console, do the following:</div>
<div>
<ol>
<li>/usr/lib/zeppelin/bin/zeppelin-daemon.sh stop</li>
<li>/usr/lib/zeppelin/bin/zeppelin-daemon.sh start</li>
</ol>
<div>
If you wish to execute the scripts in zepplin account, which has a nologin shell. Execute following instead:</div>
</div>
<div>
<ol>
<li>sudo -s /bin/bash -c '/usr/lib/zeppelin/bin/zeppelin-daemon.sh stop' zeppelin</li>
<li>sudo -s /bin/bash -c '/usr/lib/zeppelin/bin/zeppelin-daemon.sh start' zeppelin</li>
</ol>
</div>
If you encounter this Java connection error: <i>java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method)</i>, it's probably because Zeppelin starts the spark interpreter in a different process.<br />
<br />
<ol>
<li>Edit <u>/etc/spark/conf/spark-defaults.conf</u></li>
<li>Comment off the following line and restart Zeppelin</li>
</ol>
<br />
<pre style="background-color: #eff0f1; border: 0px; color: #242729; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-size: 13px; margin-bottom: 1em; max-height: 600px; overflow: auto; padding: 5px; width: auto; word-wrap: normal;"><code style="border: 0px; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; margin: 0px; padding: 0px; white-space: inherit;">#spark.driver.memory 5g</code></pre>
<div>
Reference: http://stackoverflow.com/questions/32735645/hello-world-in-zeppelin-failed</div>
<br />
<div>
<ol></ol>
</div>
samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com338tag:blogger.com,1999:blog-641042077681106765.post-89782267404978494452016-05-31T20:55:00.003+08:002016-08-07T22:28:00.514+08:00Multiple JSON Configurations for Amazon EMR cluster<span style="background-color: white; font-family: "helvetica neue" , "helvetica" , "arial" , sans-serif; font-size: 14px; line-height: 20px;"><span style="color: #212121;">To use multiple JSON configurations when you launch the new Amazon EMR cluster, I want to configure Spark to use dynamic allocation of executors and store Zeppelin notebook on S3 storage. Rename the bold red below according to your S3 bucket location. In the following example, create the folder </span><b style="color: #212121;">'/user/notebook'</b><span style="color: #212121;"> under </span><b><span style="color: red;">your-s3-bucket</span></b><span style="color: #212121;">. You'll see new note.json under the S3 folder, as you create new Zeppelin notebooks.</span></span><br />
<pre class="jive-pre" style="background: url("https://dn1lvoe5mgzdm.cloudfront.net/images/quote-background-1.gif") rgb(247, 247, 247); border-color: rgb(238, 238, 238) rgb(238, 238, 238) rgb(238, 238, 238) rgb(204, 204, 204); border-style: solid; border-width: 1px 1px 1px 2px; margin: 1em 2em 0.5em; overflow: auto; padding: 10px 10px 10px 1em; width: 600px; word-wrap: break-word;"><pre style="background: rgb(238, 238, 238); border-radius: 4px; border: 1px solid rgb(204, 204, 204); font-family: "andale mono", "lucida console", monospace; font-size: 13px; font-stretch: normal; line-height: 19.5px; margin-bottom: 1.5em; margin-top: 1.5em; padding: 0.75em 1em; vertical-align: baseline; white-space: pre-wrap;"><span style="color: #333333;">[
{
"classification":"spark-defaults",
"properties": {
"spark.serializer":"org.apache.spark.serializer.KryoSerializer",
"spark.dynamicAllocation.enabled":"true"},
"configurations":[]
},
{
"configurations":[
{
"classification":"export",
"properties":{
"ZEPPELIN_NOTEBOOK_S3_BUCKET":"</span><b><span style="color: red;">your-s3-bucket</span></b><span style="color: #333333;">",
"ZEPPELIN_NOTEBOOK_STORAGE":"org.apache.zeppelin.notebook.repo.S3NotebookRepo",
"ZEPPELIN_NOTEBOOK_USER":"</span><span style="color: red;">user</span><span style="color: #333333;">"</span><cps style="color: #333333;">
}
}
],
"classification":"zeppelin-env",
"properties":{
}
}
]</cps></pre>
</pre>
samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com11tag:blogger.com,1999:blog-641042077681106765.post-59957995242746979592016-02-27T22:17:00.002+08:002016-02-27T22:17:45.130+08:00Run Node.JS within Sublime Text editorStep 1: Open "Sublime Text 2" editor<br />Step 2: Tools -> Build System -> New Build System<br />Step 3: New tab appears. Replace the content with the following lines.<br />
<br />
{<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>"cmd": ["node", "$file", "$file_base_name"],<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>"working_dir": "${project_path:${folder}}",<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>"selector": "*.js"<br />
}<br />
<br />
Step 4: Save the file and rename it with "NodeJS.sublime-build".<br />
Step 5: Select "Tools -> Build System -> NodeJS"<br />
Step 6: Go to your source program and press "Ctrl-B" to run code.samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com4tag:blogger.com,1999:blog-641042077681106765.post-4515654266255689712016-02-03T14:49:00.000+08:002016-02-03T15:02:39.210+08:00Connect to WS2012 WSUS Internal DatabaseTo connect to the Windows Internal Database (WID) WSUS, install SQL Management Studio and connect to the server using:<br />
<b><br /></b>
<br />
<blockquote class="tr_bq">
<b>\\.\pipe\<span style="color: red;">MICROSOFT##WID</span>\tsql\query</b></blockquote>
<br />
You may confirm the SQL instance name in red against the startup service description using services.msc.<br />
<br />
As there is a constant error of "WSUS server is still processing a previous configuration change", this is what I need to execute on the database instance:<br />
<br />
<blockquote class="tr_bq">
<b>USE </b>SUSDB; </blockquote>
<blockquote class="tr_bq">
<b>UPDATE </b>tbSingletonData
<b>SET </b>ResetStateMachineNeeded = 0</blockquote>
samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com1tag:blogger.com,1999:blog-641042077681106765.post-75112519727009346332015-08-28T17:42:00.000+08:002015-08-28T17:42:10.918+08:00Develop Apache Spark Apps with IntelliJ IDEA on Windows OSI've posted an LinkedIn article on "<a href="http://www.linkedin.com/pulse/develop-apache-spark-apps-intellij-idea-windows-os-samuel-yee?trk=prof-post" target="_blank">Develop Apache Spark Apps with IntelliJ IDEA on Windows OS</a>".samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com1tag:blogger.com,1999:blog-641042077681106765.post-28547362572762699102015-08-24T12:38:00.000+08:002015-08-24T12:38:30.761+08:00Making Sense of "User-Based Recommender in 5 minutes"Have you wondered how Amazon recommend new items to you? This is an example of Machine Learning implementation, which is a type of Artificial Intelligence. I have followed an introductory example of Apache Mahout and shared this on <a href="https://www.linkedin.com/pulse/making-sense-user-based-recommender-5-minutes-samuel-yee?trk=prof-post" target="_blank">LinkedIn</a>.samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com0tag:blogger.com,1999:blog-641042077681106765.post-8580193308955575922015-08-18T17:15:00.003+08:002015-08-18T17:15:28.703+08:00Introducing Apache Pig on Amazon EMRI've just published "<a href="https://www.linkedin.com/pulse/introducing-apache-pig-amazon-emr-samuel-yee?published=u" target="_blank">Introducing Apache Pig on Amazon EMR</a>" on LinkedIn.samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com0tag:blogger.com,1999:blog-641042077681106765.post-80655140781800906472015-08-05T10:23:00.000+08:002015-08-13T10:23:53.174+08:00Develop Hadoop Apps with HortonWorks and IntelliJ IDEA on Windows OSI have published "<a href="https://www.linkedin.com/pulse/develop-hadoop-apps-hortonworks-intellij-windows-os-x64-samuel-yee?trk=prof-post" target="_blank">Develop Hadoop Apps with HortonWorks and IntelliJ IDEA on Windows OS</a>" on LinkedIn.samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com0tag:blogger.com,1999:blog-641042077681106765.post-74221084226667617852015-06-11T10:20:00.000+08:002015-08-13T10:21:13.709+08:00Collecting #SEAGames2015 Tweets for Sentiment Analysis on AWSI have published "<a href="https://www.linkedin.com/pulse/collecting-seagames2015-tweets-sentiment-analysis-aws-samuel-yee?trk=prof-post" target="_blank">Collecting #SEAGames2015 Tweets for Sentiment Analysis on AWS</a>" on LinkedIn.samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com0tag:blogger.com,1999:blog-641042077681106765.post-8684038121131180562015-06-08T10:18:00.000+08:002015-08-13T10:18:56.293+08:00Getting Started with Data Visualization using TableauI have published an article on LinkedIn - "<a href="https://www.linkedin.com/pulse/getting-started-data-visualization-using-tableau-samuel-yee?trk=prof-post" target="_blank">Getting Started with Data Visualization using Tableau</a>"samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com0tag:blogger.com,1999:blog-641042077681106765.post-48224272286142383502015-05-26T10:15:00.000+08:002015-08-13T10:16:23.163+08:00Getting Started with Amazon Elastic MapReduce for Big DataI have published an LinkedIn article on "<a href="https://www.linkedin.com/pulse/getting-started-amazon-elastic-mapreduce-samuel-yee?trk=mp-reader-card" target="_blank">Getting Started with Amazon Elastic MapReduce for Big Data</a>".samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com1tag:blogger.com,1999:blog-641042077681106765.post-68820485656012541322015-04-14T10:58:00.001+08:002015-04-14T11:07:16.553+08:00Let us overcome the fear & anxiety of Cloud<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-hQpFgWz5fsM/VSyCKaxvtsI/AAAAAAAADmk/G262Cg_6FlI/s1600/Cloud-adoption.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-hQpFgWz5fsM/VSyCKaxvtsI/AAAAAAAADmk/G262Cg_6FlI/s1600/Cloud-adoption.jpg" height="320" width="316" /></a></div>
<br />
I have come across this article "<a href="http://www.business2community.com/cloud-computing/employing-cloud-2015-01135838#xPSaDomKg8ET3ysO.99" target="_blank">Employing Cloud Into 2015</a>" that predicts this year is when cloud would go mainstream. These are the statements that I agree most with:<br />
<blockquote class="tr_bq">
"Enterprises that embrace cloud computing reduce the amount of IT time and budget devoted to legacy systems and routine upgrades, which then increases the time and budget they have for more innovative projects. When IT innovation happens, business innovation is reached, which then supports job creation." – IDC Chief Research Officer John F. Gantz</blockquote>
<blockquote class="tr_bq">
“For most organizations, cloud computing should be a no-brainer, given its ability to increase IT innovation and flexibility, lower capital costs, and help generate revenues that are multiples of spending,” said John F. Gantz, chief research officer and senior vice president at IDC.</blockquote>
So, is Y2015 the year of cloud adoption? In fact, cloud has already gone mainstream into our personal lives since many years ago. Look at how cloud companies like Facebook, Google and Amazon change the way we live, play, communicate, buy and sell. Yet, many still remain indifferent in our workplace when come to cloud adoption. It's not about data security or reliability anymore. It has been proven how Amazon and Google run and build data centers with much higher standards than most of ours - be it reliability, security and even cost-efficiency. So what's stopping cloud adoption at our workplace, even though all of us have been living in cloud for such a long time? I believe many of us, especially the younger GenY-ers, probably can't live without it (besides their smartphones).<br />
<br />
I suspect it's ALL about fears and job insecurity. For long, we have been reading about how cloud can automate and make things happen faster and cheaper. Unspoken fearful questions might creep in, "Would I still be needed?", "Would I be replaced?", "How could I justify for my next performance review and promotion?". Instead of finding out more about cloud adoption, many simply run away from it (and eventually be left behind).<br />
<br />
Yes, cloud would certainly take away mundane and boring jobs today. Tomorrow, it will bring in new exciting innovative jobs that would propel us ahead of (or at least keep up to) our competitors. To smooth this transition that is inevitable, we should play our parts as cloud evangelists to help bring enlightenment to our friends and colleagues. Knowledge is the ultimate anecdote to overcome fear and anxiety. Signing up for free tier services provided by cloud providers, like Amazon Web Services and Microsoft Azure, is the first step toward cloud enlightenment.<br />
<div>
<br /></div>
samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com0tag:blogger.com,1999:blog-641042077681106765.post-85705947121558268372015-01-21T14:45:00.002+08:002015-04-14T10:58:56.746+08:00Installing OwnCloud on AWSI came across this blog post that shows how to install <a href="https://awssolutions.wordpress.com/2013/07/19/installing-owncloud/" target="_blank">OwnCloud on AWS</a>. The post contains several typo errors that should be easy to spot out.samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com2tag:blogger.com,1999:blog-641042077681106765.post-79478357867295815272014-11-11T16:09:00.000+08:002014-11-12T11:03:11.873+08:00First experience using AWS Virtual Private Cloud<div style="text-align: justify;">
All the while, Amazon Web Services (AWS) - the leading public cloud provider - has been advocating for the demise of On-Premise Private Cloud. Its nearest alternative offering is Virtual Private Cloud (VPC) where you could really build one anytime, anywhere. Using its free usage tier, I've built a base VPC with two EC2 instances (or VMs) as depicted below.</div>
<div class="separator" style="clear: both; text-align: justify;">
<a href="http://2.bp.blogspot.com/-JFYonVAfM4Y/VGG-ti68rwI/AAAAAAAAB5I/ZI4ZxdYJPbY/s1600/AWS-VPC-Blog.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-JFYonVAfM4Y/VGG-ti68rwI/AAAAAAAAB5I/ZI4ZxdYJPbY/s1600/AWS-VPC-Blog.jpg" height="536" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
To learn AWS, we have to understand its terminology:</div>
<br />
<ol>
<li style="text-align: justify;"><b>What's EC2 Instance?</b> It's Virtual Machine.</li>
<li style="text-align: justify;"><b>What's VPC?</b> Virtual network on cloud where you can create multiple IP subnets on it. EC2 instances may be hosted on a VPC.</li>
<li style="text-align: justify;"><b>What's Security Group?</b> Think of it like a L2 firewall where you can configure the network access rules e.g. only allow HTTPS to the public Web instance from Internet etc. It is associated to one or more instances.</li>
<li style="text-align: justify;"><b>What's Subnet</b>? The usual IP subnet that we knows of. A VPC is made up of one or more subnets. You can configure which subnet is public facing and which are not. In my example, 10.10.1.0/24 is public facing and 10.10.4.0/24 to host my internal instances.</li>
<li style="text-align: justify;"><b>What's Network ACL</b>? Think of it like the usual network ACL applied to router interfaces. The ACL is stateless, so you've to define both inbound and outbound for a particular traffic. It can be used to complement the Security Group. For example, allow inbound TCP 443 to the subnet that hosts the above Web instance.</li>
<li style="text-align: justify;"><b>What's Elastic IP (EIP)?</b> A public IP assigned to a public facing instance, although only private IP is assigned physically on its NIC. Think of it like an NAT address on the invisible Internet gateway.</li>
</ol>
<div style="text-align: justify;">
To begin with free trial:</div>
<br />
<ol>
<li style="text-align: justify;">Of course, create an <a href="http://aws.amazon.com/" target="_blank">AWS account</a> using your credit card. Don't worry, AWS won't charge anything to your card, as long as you stay within the free usage tier. You can enable bills monitoring if you're concern that you would exceed the free tier limit. As for me, the ultimate backstopper is to make friend with the extremely friendly AWS account managers.</li>
<li style="text-align: justify;">Start with the <a href="http://aws.amazon.com/quickstart/" target="_blank">AWS Quick Start guides</a>, especially the RD Gateway guide.</li>
<li style="text-align: justify;">Create a VPC with 2 subnets - one public facing (i.e. RD Gateway for remote admin) and another private subnet to host the internal instances.</li>
<li style="text-align: justify;">Launch new instances with a wide range of Amazon Machine Image (AMI) templates to select, including various Windows Server and Linux OSes.</li>
<li style="text-align: justify;">Configure the Security Group to allow inbound RDP TCP 3389 for the initial setup of RD Gateway instance.</li>
<li style="text-align: justify;">After the RD Gateway is successfully setup, you can tighten network security by allowing only HTTPS traffic.</li>
</ol>
<br />
<div style="text-align: justify;">
So far, the usage experience on AWS is good, as though I’m working on my own private cloud. The free SDN feature provided by AWS is also almost as agile and flexible as the VMWare NSX that I've recently experimented with. I’m also impressed by the AWS powershell supports embedded in the Windows template. Most importantly, all the AWS features are well documented. The only ‘complaint’ so far is the relative slow loading of html AWS documentation (probably not hosted/cached in Singapore?)</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
But can AWS really replace all on-premise private cloud networks? It definitely hold promises due to its great elasticity and flexibility. The next challenge depends on how fast its metering jump, whereas in private cloud world where metering is rarely looked at (lest even use). Much like the debate of whether it's more economical of hiring taxi daily vs owning a car, which cost can be astronomical in Singapore. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com7tag:blogger.com,1999:blog-641042077681106765.post-34465676555432711722014-11-01T13:50:00.002+08:002014-11-01T13:50:32.400+08:00My latest DIY ComputerMy latest DIY computer: i7 CPU, 16GB RAM, full SSD drive, Nvidia GTX 650 GPU, Gold-class PSU and a brand new LED monitor. In a blink of eye, Windows and most apps will fire up instantly without delay. All in for just SGD 1,600. Realised my dream to have a home "Data Center in a Box" by enabling Hyper-V for both entertainment and R&D purposes. I can bet that this monster can run faster than all the 5-figure and 6-figure servers at my workplace.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-0jJLyRnrpt4/VFR0f132POI/AAAAAAAAB3c/Vg7lfRRkWxY/s1600/DIYComp.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-0jJLyRnrpt4/VFR0f132POI/AAAAAAAAB3c/Vg7lfRRkWxY/s1600/DIYComp.jpg" height="191" width="320" /></a></div>
<br />
As for Cisco routers simulation, I'll need VMWare ESXi for the CSR1000V. I'll work on ESXi USB stick for alternate boot, using my old laptop to vSphere in.samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com0tag:blogger.com,1999:blog-641042077681106765.post-80086315158928567662014-10-24T15:47:00.000+08:002014-10-24T16:18:18.777+08:00Trying out VMWare NSX Hands-On-Lab (HOL)Just did my first lab on <a href="http://www.projectnee.com/HOL/catalogs/" target="_blank">VMWare NSX Hands-On-Lab</a> on network virtualization. The task is to create a logical L2 network between 2 VMs, even though they could be separated by underlying L3 physical network and even residing on different clusters. Finally, the lab will bridge the VXLAN logical switch to VLAN 100 on production network.<br />
<br />
Let's understand the key NSX components:<br />
<ol>
<li><b>NSX Manager</b> is the centralized network management component of NSX, and is installed as a <u>virtual appliance</u>. It provides an aggregated system view.</li>
<li><b>NSX controller</b> is the central control point for all logical switches within a network and maintains information of all virtual machines, hosts, logical switches, and VXLANs. The controller is running as a VM. For redundancy, a second controller can also serve as a standby VM. The controller supports two new logical switch control plane modes, Unicast and Hybrid. These modes decouple NSX from the physical network. VXLANs no longer require the physical network to support multicast.</li>
<li><b>NSX Edge</b> provides network edge security and gateway services to isolate a virtualized network and to bridge or route to physical network. You can install NSX edge either as a logical router or a services gateway.</li>
<li><b>NSX vSwitch</b> replaces the default Virtual Distributed Switch (VDS) on the kernel mode hypervisor on each host.</li>
</ol>
<div>
I won't go through the detailed step-by-step. Rather, I would highlight the high level steps to serve better understanding on the deployment scenario.</div>
<div>
<br /></div>
<div>
Step 0: Preparing the network and clusters. There are 3 clusters of hosts. Compute A and B are 2 clusters are meant for hosting VMs. NSX vSwitch resides on all hypervisors. Other NSX components mentioned above reside on the "Management and Edge Cluster".</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-bNYP6yAGgt4/VEoAfiX2TXI/AAAAAAAABz0/FeF8---bShA/s1600/Step0%2B-%2BVDS.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-bNYP6yAGgt4/VEoAfiX2TXI/AAAAAAAABz0/FeF8---bShA/s1600/Step0%2B-%2BVDS.jpg" height="397" width="640" /></a></div>
<div>
<br /></div>
<div>
Step 1: Enable VXLAN Tunnel End Points (VTEPs) and VXLAN using vSphere client.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-MOB0il6fqNM/VEoAfo9ppsI/AAAAAAAABz4/WQ2zsA3QDYU/s1600/Step1%2B-%2BVXLAN.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-MOB0il6fqNM/VEoAfo9ppsI/AAAAAAAABz4/WQ2zsA3QDYU/s1600/Step1%2B-%2BVXLAN.jpg" height="413" width="640" /></a></div>
<div>
<br /></div>
<div>
Step 2: Create a VXLAN Transport Zone spanning the 3 clusters</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-GgdQYhyYlSY/VEoAgYoaaDI/AAAAAAAABz8/W51Nt5c7s1w/s1600/Step2%2B-%2BTRANSPORT%2BZONE.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-GgdQYhyYlSY/VEoAgYoaaDI/AAAAAAAABz8/W51Nt5c7s1w/s1600/Step2%2B-%2BTRANSPORT%2BZONE.jpg" height="416" width="640" /></a></div>
<div>
<br /></div>
<div>
Step 3: Create a logical switch and attach it to an NSX Edge. The Edge gateway has an interface of 192.168.100.1 connecting to the transport zone. Note the new L2 logical network (172.16.40.0/24) created in green.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-ToSTaheuW24/VEoAg5USLuI/AAAAAAAAB0E/qa44RaHQ1YE/s1600/Step3%2B-%2BAdd%2BLogical%2BSwitch%2Bto%2BNSX%2BEdge%2BGW.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-ToSTaheuW24/VEoAg5USLuI/AAAAAAAAB0E/qa44RaHQ1YE/s1600/Step3%2B-%2BAdd%2BLogical%2BSwitch%2Bto%2BNSX%2BEdge%2BGW.jpg" height="416" width="640" /></a></div>
<div>
<br /></div>
<div>
Step 4: Add two Web VMs and their vNICs to the new logical network as shown below. Both static IP and DHCP should work fine on the VMs. Test connectivity between both VMs.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-1_W-Xe2lirk/VEoAhJz92dI/AAAAAAAAB0M/2Hb3p8qo-uU/s1600/Step4%2B-%2BAdd%2BVMs%2Bto%2BLogical%2BSwitch.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-1_W-Xe2lirk/VEoAhJz92dI/AAAAAAAAB0M/2Hb3p8qo-uU/s1600/Step4%2B-%2BAdd%2BVMs%2Bto%2BLogical%2BSwitch.jpg" height="412" width="640" /></a></div>
<div>
<br /></div>
<div>
Step 5: Bridging the logical switch to the physical network via NSX Logical Router. In this case, the VXLAN is bridged to VLAN 100 on the production network.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-wLT_mvAvTkU/VEoAhO_R6RI/AAAAAAAAB0I/B67htFID-I4/s1600/Step5%2B-%2BV2P.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-wLT_mvAvTkU/VEoAhO_R6RI/AAAAAAAAB0I/B67htFID-I4/s1600/Step5%2B-%2BV2P.jpg" height="400" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com0tag:blogger.com,1999:blog-641042077681106765.post-62267617901269126522014-10-04T17:03:00.001+08:002014-10-04T17:03:40.107+08:00Windows Azure AD with your Active Directory<div style="background-color: white; border: 0px; box-sizing: border-box; color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 15.5555562973022px; line-height: 23.9999980926514px; margin-bottom: 30px; outline: 0px; padding: 0px; vertical-align: baseline;">
I've just watched a Microsoft jump-start video on how to integrate Windows Azure AD (AAD) with your on-premise AD infrastructure. By doing so, your users can experience seamless authentication experience between public Windows Azure (e.g. Office 365, Sharepoint online etc) and on-premise network. Here is the link: <a href="http://channel9.msdn.com/Series/Windows-Azure-for-IT-Professionals/Windows-Azure-for-IT-Professionals-05-AD-to-Windows-Azure-AD" rel="nofollow" style="border: 0px; box-sizing: border-box; color: #7b539d; font-family: inherit; font-size: 15.5555562973022px; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;" target="_blank">AD to Windows Azure AD</a>.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 15.5555562973022px; line-height: 23.9999980926514px; margin-bottom: 30px; outline: 0px; padding: 0px; vertical-align: baseline;">
In summary, there are 3 possible options:</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 15.5555562973022px; line-height: 23.9999980926514px; margin-bottom: 30px; outline: 0px; padding: 0px; vertical-align: baseline;">
1) <strong style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 15.5555562973022px; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">No integration</strong>. Users logon to Azure and on-premise AD separately with different sets of credential.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 15.5555562973022px; line-height: 23.9999980926514px; margin-bottom: 30px; outline: 0px; padding: 0px; vertical-align: baseline;">
2) <strong style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 15.5555562973022px; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Directory sync (DirSync) only</strong>: On-premise AD user accounts and password hashes are synced to Azure. Users logon to both using same set of credential. No Single Sign-On (SSO) between AD and AAD. In other words, users have to authenticate twice, even though they may use the same set of user ids and passwords.</div>
<div style="background-color: white; border: 0px; box-sizing: border-box; color: #4d4f51; font-family: Helvetica, Arial, sans-serif; font-size: 15.5555562973022px; line-height: 23.9999980926514px; margin-bottom: 30px; outline: 0px; padding: 0px; vertical-align: baseline;">
3) <strong style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: 15.5555562973022px; font-style: inherit; font-variant: inherit; line-height: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">AD Federation (ADFS with DirSync)</strong>: AD user objects (but no password hash) are synced to Azure. Establish one-way federated trust (i.e. Azure trusts your AD). This option supports SSO and even smart card authentication.</div>
samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com7tag:blogger.com,1999:blog-641042077681106765.post-58239871974913121332014-09-03T17:30:00.000+08:002014-09-03T17:31:55.283+08:00Active directory or sysvol is not accessible on this domain controller or an object is missingI saw this error message on Group Policy Management when I did a status check on the AD replication. All domain controllers were stuck with replication in progress with their respective Sysvol "inaccessible" against the PDC emulator. I couldn't find any error events on "DFS Replication" at all - the replication just got stuck in progress.<br />
<br />
When this happens, follow the steps on <a href="http://support.microsoft.com/kb/2218556" target="_blank">How to perform an authoritative synchronization of DFSR-replicated SYSVOL</a>.samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com0tag:blogger.com,1999:blog-641042077681106765.post-6867160150880778272014-08-28T09:10:00.002+08:002014-08-28T09:11:13.573+08:00Trying out Lync 2013 DeploymentI did my first test deployment for Lync 2013 - a Skype-like application for Intranet. For quick step-by-step installation, I've followed this guide: <a href="http://www.orcsweb.com/blog/cory-granata/how-to-install-lync-server-2013-standard-edition-on-windows-server-2012/" target="_blank">How to Install Lync Server 2013 Std. Edition on Windows Server 2012</a><br />
<br />
As installing Lync server requires modifying the AD forest, I've decided to make it cross-forest i.e. Lync on resource forest. It has similar concept of Linked Mailbox in Exchange i.e. disabled user account on resource forest that map to the actual user SID on user forest. To do so, I've followed this guide: <a href="http://thecathcart.blogspot.sg/2012/07/user-enabling-in-lync-resource-forest.html" target="_blank">User Enabling in Resource Forest</a><br />
<br />
If you do not have an Exchange server on resource forest, you can simply just (on resource forest):<br />
<br />
<ol>
<li>Create a new disabled user account with same email address as the user.</li>
<li>Copy the objectSID attribute from the User Forest account to the <b><i><span style="color: blue;">msRTCSIP-OriginatorSID </span></i></b>attribute of the disabled account. You can simply do so using the "AD Users and Computers" console by enabling "Advanced Features" on the "View" menu.</li>
</ol>
samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com0tag:blogger.com,1999:blog-641042077681106765.post-30273888991054322312014-05-15T10:28:00.002+08:002014-05-15T10:37:17.646+08:00Virtualised Domain Controllers Replication IssuesI noticed virtualised domain controllers often have issues replicating new settings in Group Policy Objects. This warning message was also observed:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-AqQSOhL8s0I/U3QoKKA45PI/AAAAAAAAAsk/y0Fe_uJVbhU/s1600/paused.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-AqQSOhL8s0I/U3QoKKA45PI/AAAAAAAAAsk/y0Fe_uJVbhU/s1600/paused.jpg" height="267" width="400" /></a></div>
<br />
<blockquote class="tr_bq">
<b>Error: 9036 (Paused for backup or restore)</b></blockquote>
After reading this Technet article on <a href="http://technet.microsoft.com/pt-pt/library/virtual_active_directory_domain_controller_virtualization_hyperv(v=ws.10).aspx#backup_and_restore_considerations_for_virtualized_domain_controllers" target="_blank">backing up virtual domain controller</a>, I realised the cause was due to the snapshot back at Hyper-V level. The only supported backup method is running the backup job at the guest VM level. Since then, I've stopped backing up domain controllers at Hyper-V host level and disabled the backup integration services at VM configuration.samyeehttp://www.blogger.com/profile/00953121441659794088noreply@blogger.com0