Serverless Latency

Serverless applications really interest me.  What is the current guidance on use cases, best practices, etc..?  Anyways I stumbled upon this post and found it extremely interesting as it dives deep in to Serverless latency, its causes, and ways to work around performance issues.

https://www.tbray.org/ongoing/When/201x/2018/12/14/SF-4

Excelero a company to watch

As the technology world continues to move towards software defined everything Excelero's NVMe-oF storage platform is really interesting.  I really like the RDMA aspects of it and don't find that a significant cost or issue.  I also like the new Erasure Coding option for data protection.  It appears to be VERY similar to Storage Spaces Direct EC.  For more information read:  https://www.nextplatform.com/2018/10/31/broadening-the-appeal-of-distributed-nvm-express-storage/

Windows Kernel improvements

Really interesting read on how Microsoft has improved the kernel to take best advantage of large numbers of logical CPU's. Latest milestone is 896 cores with 1792 logical processors!!!!!!!!!!!!!!!! https://techcommunity.microsoft.com/t5/Windows-Kernel-Internals/One-Windows-Kernel/bc-p/278805#M18

Microsoft SDN Load Balancer

I really look forward to diving into Microsoft's SDN play in the future. I hope whatever company I end up working for is thinking about leveraging Hyper-v, SDN, and Storage Spaces Direct. https://blogs.technet.microsoft.com/networking/2018/10/10/notesfromthefield-slb/

New to or Thinking About Microsoft Storage Spaces Direct?

For those of you who are interested in moving off traditional compute/SAN converged infrastructure you are at the right place. Storage Spaces Direct provides an extremely high-performance hyper-converged infrastructure (HCI) that can be tuned and used in a myriad of different ways.

Over the course of the last 20 months, we at Bradley rolled out 10 different clusters of Storage Spaces Direct with use cases spanning archival backups, remote office/branch office VM, and high-performance Tier 0 VM workloads.

Here are a few things I have learned during this time.

Focus on a ‘keep it simple model’ and create a modular design. In my opinion, Microsoft has created a “choose your own adventure” model for its HCI offering. It’s an extremely customizable and feature-rich product and therefore CAN be complex if that is what you want. As I’ve architected/implemented our Storage Spaces Direct infrastructure I’ve found that a focus on modularity and simplicity is the key to a successful implementation. Break up your installation into chunks/modules, for example, Windows Server installs, Hyper-V/failover cluster, networking, Storage Spaces Direct. This will allow you to test each section/feature before implementing the next and dramatically reduce any troubleshooting you may have to perform. I also recommend for your initial deployments that you follow well laid out guides from Microsoft, Mellanox, and others to create your first clusters. As you become comfortable with the product you’ll be able to grow its capabilities in a linear and modular fashion on an as-needed basis. For an example of the full extent of what Storage Spaces Direct can provide, look no further than Azure, Microsoft’s public cloud-based platform, as it runs on Hyper-V with Storage Spaces Direct. 

Hardware is critical. What I mean by this is that your Storage Spaces Direct experience will only be as good as the hardware you choose to run it on. One of the major selling points of Storage Spaces Direct, in my opinion, is its use of off-the-shelf components. For the best experience make sure the components you choose to conform to the Windows Server Catalog and have the SDDC Premium/Standard certification. We looked at many different vendors offering Microsoft-validated designs during our proof of concept phase and found DataON to provide the best hardware at the best price. One of DataON’s recommendations was to use Intel Optane NVMe SSDs for the cache tier and that delivered a vast improvement in latency and IOPS. Finally, make sure you size your physical nodes’ CPU and RAM properly. While small and getting smaller all the time, Storage Spaces Direct and the ReFS file system does have some minimal overhead associated with its use. DataON can be instrumental in helping you size your nodes correctly. 

Take the time to build out a deployment PowerShell script that you can re-use. This is the perfect way to ensure that you are configuring each node of your cluster with the same features, functionality, and configuration in a repeatable fashion. Also, as a side benefit, by using a script you are automatically documenting your implementation at the same time. There are lots of resources in the blogosphere on examples of how to leverage PowerShell cmdlets to install all aspects of Windows Server, Hyper-V, and Storage Spaces Direct. On that note, GitHub is your friend. Microsoft has a phenomenal resource for Storage Spaces Direct help located here. Also, the PowerShell script I created to automate my Storage Spaces Direct cluster implementation is located here. 

Software Defined Networking (SDN). A big part of Hyper-V and Storage Spaces Direct revolves around the use of SDN. It is very important to layout an IP/Subnet and VLAN scheme before you install Storage Spaces Direct. Some features (vSwitches come to mind) can only be configured during creation and not after. A MUST read resource is provided by Microsoft here. Another item to consider is the costs/features of “white box” switches and Cisco/Arista/Juniper. I love Mellanox’s SN2700 Series switch coupled with Mellanox adapters. It is a fraction of the cost of a Cisco solution, is highly performing at 100Gb per port, and makes configuring RoCE RDMA solution very simple. Community support for configuration designs/questions is also really good at Mellanox. 

For management of your Storage Spaces Direct environment, I personally really like to use Failover Cluster Manager for most of my VM management tasks. As Windows Admin Center features continue to grow this will become more and more my go-to management tool. We also use System Center Virtual Machine Manager. I personally would only recommend using SCVMM if you have a Role Based Access requirement at your company as managing Hyper-V via Failover Cluster Manager or Hyper-V manager requires Administrator rights.


The best advice I have ever received in my 20 years of being in IT is to “make time every day to read the latest tech news and blogs.” This will help you to stay current, learn new ways of doing things, and discover pitfalls to avoid.

Here are a few sites I follow that provide information on Storage Spaces Direct and troubleshooting advice:
https://storagespacesdirect.slack.com
http://www.aidanfinn.com/
https://www.altaro.com/hyper-v/
https://www.thomasmaurer.ch/
https://blogs.technet.microsoft.com/virtualization/
https://blog.workinghardinit.work/

Twitter is also a great resource!

I hope you find the above information useful. Storage Spaces Direct coupled with DataON hardware is an amazing alternative to monolithic designs of storage and compute most companies are used to. I highly encourage companies of all sizes to try it for themselves.




http://blog.dataonstorage.com/2018/09/13/it-hero-blog-getting-started-with-storage-spaces-direct/

My HCI build of Microsoft Storage Spaces Direct featured as DataOn's IT Hero

http://blog.dataonstorage.com/2018/09/13/it-hero-blog-getting-started-with-storage-spaces-direct/

Oldie but goodie :)

Very few Physical to Virtual conversions going on now a days but nice to keep this in the back of your mind if you have a need to convert a physical machine to a Hyper-V Generation 2 VM:  https://blog.workinghardinit.work/2016/09/08/disk2vhd-on-a-generation-2-vm-results-in-an-unbootable-vhdx/

Choosing a platform is about picking the best services to deliver the application

https://blog.architecting.it/enterprise-class-public-cloud/?utm_source=rss-feed&utm_medium=rss&utm_campaign=feed

Great information for all you Hyper-V fans!

http://www.checkyourlogs.net/?p=56083

Very good read by Ethan Banks

The Most Fragile Engineers I Know

by Ethan Banks

This tweet crossed my Twitter timeline recently. I’m confident it’s not actually by the legendary investor Warren Buffett because it’s not a verified account, and the last name is spelled wrong. Besides, I suspect Mr. Buffett has better things to do than fuss with social media--empire building must be quite time consuming.

Even so, I liked the sentiments expressed.

I’ve decided to make my own list from an IT engineering perspective. And instead of listing the smartest traits, I’ll explore the characteristics of the most fragile.

The most fragile engineers I know:

1. Are easily offended. Engineers I’ve met with fragile egos lack confidence and are offended by everything. New technology. Having to integrate with other tech silos. Having to make design compromises. Criticism. They are argumentative, angry, and hypercritical of others, describing them as “idiots,” “morons,” and other vociferous denigrations. And since managers (senior executives especially) are obviously useless, every managerial directive is a new source of outrage and offense.
   
2. Talk more than they listen. Fragile engineers need to prove to everyone that they know everything. When explaining a problem to them, they cut you off as if they already know what to do about your issue. They think they know better than the business stakeholders, and like to implement technology instead of solve problems.
   
3. Enjoy arguing. The most fragile engineers I’ve worked with are contradictory people. Their perspective is rarely “we can figure out a way to make this work.” Instead, they want to prove why the new thing can’t work, didn’t work the last time, and won’t ever work in the future. They are a barrier and not an enabler.
   
4. Never admit when they are wrong. Fragile engineers are never wrong. They never make a configuration mistake. They never unplug the wrong cable. They never shut down the interface that cuts them off from the management plane of an entire campus. They never power down the wrong router in the rack. They never make a bad recommendation. I mean, they do all of these things. We all have. But fragile engineers never admit their screw-ups, covering up with ambiguity. Truly evil engineers will blame others.
   
5. Doggedly hold an opinion. When confronted with evidence, they will contradict and fight despite what the data says. These folks seem to think that changing an opinion is some admission of inadequacy, and fragile egos can’t tolerate that. Facts be damned.
   
6. Keep smart people away. Fragile engineers are threatened by more skilled engineers, and will do their best to keep them away. After all, a skilled engineer might make them look bad, and that just won’t do. It’s not about growing into something better. It’s about protecting the existing situation.
   
7. Force others to accept their perspective. The fragile engineer doesn’t want to hear other points of view. Why bother when one’s own point of view is clearly the only correct way to think? For these folks, the best way forward is to convince everyone else of their idiocy, and wrestle them to the ground with their opinion.

After more than two decades of working with technology, I hope I’m more mature and less fragile. Writing my contrary points above, I found myself wincing at a career’s worth of memories tied to such people.

A fragile ego can steamroll relationships. The data center rack you stand up today is the one you’ll tear back out eventually. It’s not that important that you won the argument about the cable coloring scheme.

Interpersonal relationships have the potential to last much longer, and can make you better in the process. They are worth investing in. However, you need to be vulnerable before that growth can occur. Vulnerability isn’t fragility--it’s admitting you don’t know everything.

Vulnerability is a sign of maturity.

Good read on latest server hardware sales

https://www.nextplatform.com/2018/09/06/are-hyperscalers-and-clouds-builders-smashing-up-an-it-pangea/

Really Good technical compare and contrast.

Language is NSFW but the technical content is extremely impressive:

VMware vSan Performance:  http://www.cultofanarchy.org/vmware-vsan-greatin-fucking-things-re-investigating-vmware-vsan-4-node-cluster-performance/

Microsoft Storage Spaces Direct:  http://www.cultofanarchy.org/microsoft-storage-spaces-direct-s2d-surprisingly-good-job/

Nutanix:  http://www.cultofanarchy.org/perfomance-test-nutanix-4-node-cluste/

Patching S2D clusters using CAU (MUST READ)

https://bcthomas.com/2018/09/you-need-to-change-the-way-you-patch-s2d-clusters/

Operations Jobs in the future.

Topic near and dear to my heart:  https://www.enterprisetech.com/2018/09/05/no-operations-isnt-going-anywhere-but-its-going-to-look-different/?utm_source=rss&utm_medium=rss&utm_campaign=no-operations-isnt-going-anywhere-but-its-going-to-look-different

Really looking forward to Server 2019 SDN Capabilities

https://blogs.technet.microsoft.com/networking/2018/08/29/sdnsecurity/

Test Microsoft SDN using Nested Hyper-V

https://cloudmechanic.net/2018/08/29/deploying-a-sdnv2-lab-on-a-single-host-using-nested-hyper-v/

Good article on setting Visual Studio Code for Powershell development

http://blog.mpecsinc.ca/2018/08/powershell-paradise-installing-and.html

Intel Optane and Micron Split

I have to agree with the Register article on this.  Based on personal experience with a 6 node Hyper-V 2016 cluster with Optane S2D cache the performance addition is NOT worth the exorbitant price.

https://www.theregister.co.uk/2018/07/18/3d_xpoint_issues_and_micron_intel_split/

Good read on current state of Microsoft Containers

https://www.nextplatform.com/2018/07/13/microsofts-container-strategy-continues-to-evolve/

Facinating article on Flash Storage, Block Size, and Memory requirements

https://blog.architecting.it/flash-storage-dram-curve/?utm_source=rss-feed&utm_medium=rss&utm_campaign=feed

Server 2019 Build 17692 Legacy MGMT capabilitites

AWESOME!!!

b

Additional Download: Server Core App Compatibility Features on Demand

While not contained within the main bits of the build yet, this has to be one of the coolest features we’ve seen added. Insider preview build 17692 includes the ability to install a Feature on Demand package that will add several of the main Windows Server Management tools that we’ve come to rely on over the years. All of this on Server Core, and without the additional requirements and bloat from the full GUI installation!

Current tools available with this feature are:

• Performance Monitor (PerfMon.exe)
• Resource Monitor (Resmon.exe)
• Device Manager (Devmgmt.msc)
• Microsoft Management Console (mmc.exe)
• Windows PowerShell ISE (Powershell_ISE.exe)
• Failover Cluster Manager (CluAdmin.msc)
• Process Monitor (Procmon.exe) and other Sysinternals
• SQL Server Management Studio

I agree! Network Cabling

http://etherealmind.com/video-stop-fussing-network-cabling-two-beer-networking/

Just WOW

Awesome perspective of the universe:  https://youtu.be/GoW8Tf7hTGA

Azure Blob Storage Tiering

This might be worth the cost:  https://azure.microsoft.com/en-us/blog/azure-blob-storage-lifecycle-management-public-preview/

Cisco Live

hmmmmmm:  http://packetpushers.net/3-unspoken-messages-chuck-robbins-cisco-live-keynote/

Storage: Where did the IOPS go?

Crazy interesting read:  https://blog.architecting.it/avoiding-all-flash-missing-iops/?utm_source=rss-feed&utm_medium=rss&utm_campaign=feed

Mellanox and HCI

Great post on how to properly design your Network environment to best support Hyper Converged Infrastructure:  http://www.mellanox.com/blog/2018/05/ethernet-storage-fabric-part-2/

Status of the "Cloud"

Good summary on the current and future status of the "Cloud":  https://www.nextplatform.com/2018/05/18/a-hard-rains-a-gonna-fall-in-public-cloud/

Words to live by

"The complex solution is the worst solution. An enterprise solution likely requires many products from multiple vendors with different technologies and requires a range of skills to deploy and operate. The solution that requires more money, more staff and more products is more likely to fail. In short, simpler is always better but you can sound fancy when you call it “Occams Razor”.

Hmmm while I like NSX it's cost is just so exorbitant.  Will have to keep an eye on this:  https://www.theregister.co.uk/2018/05/01/vmware_cloud_network/

Yay cumulus!  https://cumulusnetworks.com/blog/announcing-cumulus-linux-3-6/

Really good article on how a person should conduct themselves in my opinion.  In my short stint as a consultant this is what I lived each day and found it best for the customer and myself:


This was pulled from a great article by Greg Ferro: http://etherealmind.com/favourite-section-anixter-blue-book/?utm_medium=feed&utm_source=feedpress.me&utm_campaign=Feed%3A+etherealmind

Really good article on current RDMA standards and configuration:  https://www.starwindsoftware.com/blog/smb-direct-the-state-of-rdma-for-use-with-smb-3-traffic-part-ii

I agree with this viewpoint.  Both the Cloud and Software Defined Datacenters have over turned all of the apple carts:

http://etherealmind.com/big-fail-likely-fail/

Cloud Storage or OnPrem, that is the question.....  At the end of the day I feel that this question is still answered with, "it depends".  I think the answer is currently largely based on:

• whether a company wants to control their data thereby keeping it in house
• operational cost control (on prem) is most important
• capital cost control (cloud) is most important

Clear as mud ;)  

https://blog.architecting.it/aws-introduces-s3-one-zone-ia/?utm_source=rss-feed&utm_medium=rss&utm_campaign=feed

Have recently started down the Mellanox hardware road with our new primary whitebox Hyper-V 2016 cluster running Storage Spaces Direct on Mellanox 2700 switches.  Super excited.  As we were going through the design and purchasing faze for our new Co-Lo infrastructure I fought to start process of migrating off of Cisco chassis based switches (7k's, 9k's) with whitebox switches.  I lost some of that battle (moving off of cisco) but succeeded in moving to fixed port based modular switches.  This article somewhat explains my viewpoint on this:  http://www.mellanox.com/blog/2018/04/why-leaf-spine-networks-taking-off/

Great option to keep up to date on latest networking news on a weekly basis.  Highly recommend them:  http://packetpushers.net

You should always ask yourself this set of questions before tweaking a nerd knob:

• What problem am I trying to solve?
• Are there better ways to solve the problem?
• What’s the worst thing that could happen if I twiddle that knob?
• Why would twiddling the nerd knob break things and what can
• I do to alleviate that (assuming I still feel it makes sense to twiddle it)?

Truly awesome!  Why hasn't this been done before?  https://www.theregister.co.uk/2017/12/18/supermicro_battery_backup_module/

The con's of complexity when using Microsegmentation COULD outway the benefits in certain environments.  I personally am a fan of securing east/west traffic but this article does highlight the difficulties associated with it: http://blog.ipspace.net/2018/02/how-useful-is-microsegmentation.html

Hmmm this is kind of neat and by trusting no-one would seem more secure and have the ability to collect robust access logs:
https://blog.cloudflare.com/introducing-cloudflare-access/

Changes for core Internet protocols!

https://www.ietfjournal.org/big-changes-ahead-for-core-internet-protocols/

SDN really excites me.  I think white/bright box hardware with a solution like Cumulus Networks just makes sense and has the potential to dramatically save companies money while also giving network operators a full featured Network OS.  Example of a dream environment: https://cumulusnetworks.com/blog/evpn-cumulus-linux-makes-cto-yell-booyah/ running on Mellanox switches.

I personally can attest to how awesome working from home is:  https://youtu.be/oiUyyZPIHyY